FTL crashes while using the web interface

Problem with Beta 5.0:
Pi-hole Version vDev (release/v5.0, v4.3.2-397-g0a70bbd) Web Interface Version vDev (release/v5.0, v4.3.2-393-g843e46d) FTL Version vDev (release/v5.0, vDev-fb01e09)

Hello.
After updating to the latest beta version there is an issue with my FTL-DNS service crashing. So far the crash only seems to happen while I am logged into the web interface. The FTL service crashes either while logging into the web interface or while switching through the different pages. It also happened on my secondary pihole server which runs a similar configuration (but with DHCP disabled). In case it is important, both piholes are configured to use unbound. DHCP on the first pihole is configured to supply clients with IP4/6 for both DNS(pihole) servers.

**Debug Token: could not be uploaded, added below (IPV6 addresses are edited due to privacy concerns)

This process collects information from your Pi-hole, and optionally uploads it to a unique and random directory on tricorder.pi-hole.net.

The intent of this script is to allow users to self-diagnose their installations.  This is accomplished by running tests against our software and providing the user with links to FAQ articles when a problem is detected.  Since we are a small team and Pi-hole has been growing steadily, it is our hope that this will help us spend more time on development.

NOTE: All log files auto-delete after 48 hours and ONLY the Pi-hole developers can access your data via the given token. We have taken these extra steps to secure your data and will work to further reduce any personal information gathered.

*** [ INITIALIZING ]
[i] 2020-02-10:16:56:57 debug log has been initialized.

*** [ INITIALIZING ] Sourcing setup variables
[i] Sourcing /etc/pihole/setupVars.conf...

*** [ DIAGNOSING ]: Core version
[i] Core: v4.3.2 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: release/v5.0 (https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#checkout)
[i] Commit: v4.3.2-397-g0a70bbd

*** [ DIAGNOSING ]: Web version
[i] Web: v4.3.2 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: release/v5.0 (https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#checkout)
[i] Commit: v4.3.2-393-g843e46d

*** [ DIAGNOSING ]: FTL version
[✓] FTL: vDev-fb01e09 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)

*** [ DIAGNOSING ]: lighttpd version
[i] 1.4.53

*** [ DIAGNOSING ]: php version
[i] 7.3.11

*** [ DIAGNOSING ]: Operating system
[✓] Raspbian GNU/Linux 10 (buster)

*** [ DIAGNOSING ]: SELinux
[i] SELinux not detected

*** [ DIAGNOSING ]: Processor
[✓] armv7l

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
   192.168.178.3/24 matches the IP found in /etc/pihole/setupVars.conf

[✓] IPv6 address(es) bound to the eth0 interface:
   2a02:*PRIVACY* matches the IP found in /etc/pihole/setupVars.conf
   fe80::*PRIVACY* does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)

   ^ Please note that you may have more than one IP address listed.
   As long as one of them is green, and it matches what is in /etc/pihole/setupVars.conf, there is no need for concern.

   The link to the FAQ is for an issue that sometimes occurs when the IPv6 address changes, which is why we check for it.

[i] Default IPv4 gateway: 192.168.178.1
   * Pinging 192.168.178.1...
[✓] Gateway responded.
[i] Default IPv6 gateway: fe80::*PRIVACY*
   * Pinging fe80::*PRIVACY*...
[✓] Gateway responded.

*** [ DIAGNOSING ]: Ports in use
[*:80] is in use by lighttpd
[*:80] is in use by lighttpd
[::1]:5353 unbound (IPv6)
127.0.0.1:5353 unbound (IPv4)
*:22 sshd (IPv4)
*:22 sshd (IPv6)

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve www.mkt6393.com via localhost (127.0.0.1)
[✗] Failed to resolve www.mkt6393.com via Pi-hole (192.168.178.3)
[✗] Failed to resolve doubleclick.com via a remote, public DNS server (8.8.8.8)

*** [ DIAGNOSING ]: Name resolution (IPv6) using a random blocked domain and a known ad-serving domain
[✗] Failed to resolve metrics.interweave.com via localhost (::1)
[✗] Failed to resolve metrics.interweave.com via Pi-hole (2a02:*PRIVACY*)
[✗] Failed to resolve doubleclick.com via a remote, public DNS server (2001:4860:4860::8888)

*** [ DIAGNOSING ]: Pi-hole processes
[✓] lighttpd daemon is active
[✓] pihole-FTL daemon is active

*** [ DIAGNOSING ]: Setup variables
    BLOCKING_ENABLED=true
    DHCP_ACTIVE=true
    DHCP_START=192.168.178.20
    DHCP_END=192.168.178.181
    DHCP_ROUTER=192.168.178.1
    DHCP_LEASETIME=168
    PIHOLE_DOMAIN=lan
    DHCP_IPv6=true
    DHCP_rapid_commit=true
    DNSMASQ_LISTENING=single
    DNS_FQDN_REQUIRED=true
    DNS_BOGUS_PRIV=true
    DNSSEC=true
    CONDITIONAL_FORWARDING=false
    PIHOLE_INTERFACE=eth0
    IPV4_ADDRESS=192.168.178.3/24
    IPV6_ADDRESS=2a02:*PRIVACY*
    PIHOLE_DNS_1=127.0.0.1#5353
    PIHOLE_DNS_2=
    QUERY_LOGGING=true
    INSTALL_WEB_SERVER=true
    INSTALL_WEB_INTERFACE=true
    LIGHTTPD_ENABLED=true

*** [ DIAGNOSING ]: Dashboard and block page
[✓] Block page X-Header: X-Pi-hole: A black hole for Internet advertisements.
[✓] Web interface X-Header: X-Pi-hole: The Pi-hole Web interface is working!

*** [ DIAGNOSING ]: Gravity List and Database
-rw-rw-r-- 1 pihole pihole 78610432 Feb 10 16:55 /etc/pihole/gravity.db
   Size (excluding blacklist): 1237645 entries

   ----- First 10 Domains -----
   localhost.localdomain
   01mspmd5yalky8.com
   0byv9mgbn0.com
   analytics.247sports.com
   www.analytics.247sports.com
   2no.co
   www.2no.co
   logitechlogitechglobal.112.2o7.net
   www.logitechlogitechglobal.112.2o7.net
   30-day-change.com


*** [ DIAGNOSING ]: Groups
   id    name                                                enabled  date_added           date_modified        description                                       
   ----  --------------------------------------------------  -------  -------------------  -------------------  --------------------------------------------------
   0     Unassociated                                        1        2020-01-19 18:21:21  2020-01-19 18:21:21                                                    

*** [ DIAGNOSING ]: Adlists
   id    address                                                                                               enabled  date_added           date_modified        comment                                           
   ----  ----------------------------------------------------------------------------------------------------  -------  -------------------  -------------------  --------------------------------------------------
   1     https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts                                      1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   2     https://mirror1.malwaredomains.com/files/justdomains                                                  1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   3     http://sysctl.org/cameleon/hosts                                                                      1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   4     https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt                                      1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   5     https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt                                            1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   6     https://hosts-file.net/ad_servers.txt                                                                 1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            
   7     https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl.txt                              1        2020-01-19 18:21:20  2020-01-19 18:21:20  Migrated from /etc/pihole/adlists.list            

*** [ DIAGNOSING ]: Adlist groups
   adli  grou
   ----  ----
   3     0   
   6     0   
   2     0   
   1     0   
   7     0   
   5     0   
   4     0   

*** [ DIAGNOSING ]: Exact whitelist

*** [ DIAGNOSING ]: Exact whitelist groups

*** [ DIAGNOSING ]: Regex whitelist

*** [ DIAGNOSING ]: Regex whitelist groups

*** [ DIAGNOSING ]: Exact blacklist

*** [ DIAGNOSING ]: Exact blacklist groups

*** [ DIAGNOSING ]: Regex blacklist

*** [ DIAGNOSING ]: Regex blacklist groups

*** [ DIAGNOSING ]: contents of /etc/pihole

-rw-r--r-- 1 root root 138 Feb 10 16:55 /etc/pihole/local.list
   192.168.178.3 raspi3
   2a02:*PRIVACY* raspi3
   192.168.178.3 pi.hole
   2a02:*PRIVACY* pi.hole

-rw-r--r-- 1 root root 234 Feb 10 14:00 /etc/pihole/logrotate
   /var/log/pihole.log {
   	su root root
   	daily
   	copytruncate
   	rotate 5
   	compress
   	delaycompress
   	notifempty
   	nomail
   }
   /var/log/pihole-FTL.log {
   	su root root
   	weekly
   	copytruncate
   	rotate 3
   	compress
   	delaycompress
   	notifempty
   	nomail
   }

*** [ DIAGNOSING ]: contents of /etc/dnsmasq.d

-rw-r--r-- 1 root root 1590 Feb 10 14:00 /etc/dnsmasq.d/01-pihole.conf
   addn-hosts=/etc/pihole/local.list
   addn-hosts=/etc/pihole/custom.list
   localise-queries
   no-resolv
   cache-size=10000
   log-queries
   log-facility=/var/log/pihole.log
   local-ttl=2
   log-async
   server=127.0.0.1
   domain-needed
   bogus-priv
   dnssec
   trust-anchor=.,19036,8,2,49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5
   trust-anchor=.,20326,8,2,E06D44B80B8F1D39A95C0B0D7C65D08458E880409BBC683457104237C7F8EC8D
   interface=eth0
   server=/use-application-dns.net/

-rw-r--r-- 1 root root 643 Feb 10 14:00 /etc/dnsmasq.d/02-pihole-dhcp.conf
   dhcp-authoritative
   dhcp-range=192.168.178.20,192.168.178.181,168h
   dhcp-option=option:router,192.168.178.1
   dhcp-leasefile=/etc/pihole/dhcp.leases
   domain=lan
   dhcp-rapid-commit
   dhcp-option=option6:dns-server,[::]
   dhcp-range=::100,::1ff,constructor:eth0,ra-names,slaac,168h
   ra-param=*,0,0

*** [ DIAGNOSING ]: contents of /etc/lighttpd

-rw-r--r-- 1 root root 3499 Feb 10 14:00 /etc/lighttpd/lighttpd.conf
   server.modules = (
   	"mod_access",
   	"mod_accesslog",
   	"mod_auth",
   	"mod_expire",
   	"mod_compress",
   	"mod_redirect",
   	"mod_setenv",
   	"mod_rewrite"
   )
   server.document-root        = "/var/www/html"
   server.error-handler-404    = "/pihole/index.php"
   server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
   server.errorlog             = "/var/log/lighttpd/error.log"
   server.pid-file             = "/var/run/lighttpd.pid"
   server.username             = "www-data"
   server.groupname            = "www-data"
   server.port                 = 80
   accesslog.filename          = "/var/log/lighttpd/access.log"
   accesslog.format            = "%{%s}t|%V|%r|%s|%b"
   index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
   url.access-deny             = ( "~", ".inc", ".md", ".yml", ".ini" )
   static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
   compress.cache-dir          = "/var/cache/lighttpd/compress/"
   compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
   mimetype.assign   = ( ".png"  => "image/png",
                         ".jpg"  => "image/jpeg",
                         ".jpeg" => "image/jpeg",
                         ".html" => "text/html",
                         ".css" => "text/css; charset=utf-8",
                         ".js" => "application/javascript",
                         ".json" => "application/json",
                         ".txt"  => "text/plain",
                         ".svg"  => "image/svg+xml" )
   include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
   include_shell "find /etc/lighttpd/conf-enabled -name '*.conf' -a ! -name 'letsencrypt.conf' -printf 'include \"%p\"
' 2>/dev/null"
   $HTTP["url"] =~ "^/admin/" {
       
       setenv.add-response-header = (
           "X-Pi-hole" => "The Pi-hole Web interface is working!",
           "X-Frame-Options" => "DENY"
       )
       $HTTP["url"] =~ ".ttf$" {
           
           setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" )
       }
   }
   $HTTP["url"] =~ "^/admin/\.(.*)" {
        url.access-deny = ("")
   }
   include_shell "cat external.conf 2>/dev/null"

*** [ DIAGNOSING ]: contents of /etc/cron.d

-rw-r--r-- 1 root root 1703 Feb 10 14:00 /etc/cron.d/pihole
   48 4   * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log
   00 00   * * *   root    PATH="$PATH:/usr/local/bin/" pihole flush once quiet
   @reboot root /usr/sbin/logrotate /etc/pihole/logrotate
   */10 *  * * *   root    PATH="$PATH:/usr/local/bin/" pihole updatechecker local
   7 15  * * *   root    PATH="$PATH:/usr/local/bin/" pihole updatechecker remote
   @reboot root    PATH="$PATH:/usr/local/bin/" pihole updatechecker remote reboot

*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 www-data www-data 1464 Feb 10 16:57 /var/log/lighttpd/error.log
   2020-02-09 00:00:39: (server.c.1759) logfiles cycled UID = 0 PID = 19791 
   2020-02-10 14:00:54: (server.c.2059) server stopped by UID = 0 PID = 1 
   2020-02-10 14:00:55: (server.c.1464) server started (lighttpd/1.4.53) 
   2020-02-10 14:00:55: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 
   2020-02-10 14:11:49: (server.c.2059) server stopped by UID = 0 PID = 1 
   2020-02-10 14:11:57: (server.c.1464) server started (lighttpd/1.4.53) 
   2020-02-10 14:11:57: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 
   2020-02-10 16:44:38: (server.c.2059) server stopped by UID = 0 PID = 1 
   2020-02-10 16:44:46: (server.c.1464) server started (lighttpd/1.4.53) 
   2020-02-10 16:44:46: (server.c.1493) WARNING: unknown config-key: alias.url (ignored) 
   2020-02-10 16:57:15: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning:  dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 192
   2020-02-10 16:57:15: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning:  array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 193
   2020-02-10 16:57:15: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning:  dns_get_record(): A temporary server error occurred. in /var/www/html/pihole/index.php on line 192
   2020-02-10 16:57:15: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning:  array_key_exists() expects parameter 2 to be array, null given in /var/www/html/pihole/index.php on line 193

*** [ DIAGNOSING ]: contents of /var/log

-rw-r--r-- 1 pihole pihole 501317 Feb 10 16:56 /var/log/pihole-FTL.log
   -----head of pihole-FTL.log------
   [2020-02-10 01:22:24.117 17723] Resizing "/FTL-per-client-regex" from 5292 to 5346
   [2020-02-10 01:23:28.490 17723] Resizing "/FTL-per-client-regex" from 5346 to 5400
   [2020-02-10 12:40:03.198 17723] Resizing "/FTL-strings" from 233472 to 237568
   [2020-02-10 14:00:49.956 17723] Shutting down...
   [2020-02-10 14:00:50.422 17723] Finished final database update
   [2020-02-10 14:00:50.429 17723] ########## FTL terminated after 1192816384.0 ms! ##########
   [2020-02-10 14:00:58.950 8283] Using log file /var/log/pihole-FTL.log
   [2020-02-10 14:00:58.950 8283] ########## FTL started! ##########
   [2020-02-10 14:00:58.950 8283] FTL branch: release/v5.0
   [2020-02-10 14:00:58.950 8283] FTL version: vDev-fb01e09
   [2020-02-10 14:00:58.950 8283] FTL commit: fb01e09
   [2020-02-10 14:00:58.950 8283] FTL date: 2020-02-10 10:20:26 +0100
   [2020-02-10 14:00:58.951 8283] FTL user: pihole
   [2020-02-10 14:00:58.951 8283] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
   [2020-02-10 14:00:58.951 8283] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
   [2020-02-10 14:00:58.951 8283]    SOCKET_LISTENING: only local
   [2020-02-10 14:00:58.951 8283]    AAAA_QUERY_ANALYSIS: Show AAAA queries
   [2020-02-10 14:00:58.951 8283]    MAXDBDAYS: max age for stored queries is 365 days
   [2020-02-10 14:00:58.951 8283]    RESOLVE_IPV6: Resolve IPv6 addresses
   [2020-02-10 14:00:58.951 8283]    RESOLVE_IPV4: Resolve IPv4 addresses
   [2020-02-10 14:00:58.951 8283]    DBINTERVAL: saving to DB file every minute
   [2020-02-10 14:00:58.951 8283]    DBFILE: Using /etc/pihole/pihole-FTL.db
   [2020-02-10 14:00:58.951 8283]    MAXLOGAGE: Importing up to 24.0 hours of log data
   [2020-02-10 14:00:58.951 8283]    PRIVACYLEVEL: Set to 0
   [2020-02-10 14:00:58.951 8283]    IGNORE_LOCALHOST: Show queries from localhost
   [2020-02-10 14:00:58.951 8283]    BLOCKINGMODE: Null IPs for blocked domains
   [2020-02-10 14:00:58.951 8283]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
   [2020-02-10 14:00:58.951 8283]    DBIMPORT: Importing history from database
   [2020-02-10 14:00:58.951 8283]    PIDFILE: Using /var/run/pihole-FTL.pid
   [2020-02-10 14:00:58.951 8283]    PORTFILE: Using /var/run/pihole-FTL.port
   [2020-02-10 14:00:58.951 8283]    SOCKETFILE: Using /var/run/pihole/FTL.sock
   [2020-02-10 14:00:58.951 8283]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
   [2020-02-10 14:00:58.951 8283]    MACVENDORDB: Using /etc/pihole/macvendor.db
   [2020-02-10 14:00:58.951 8283]    GRAVITYDB: Using /etc/pihole/gravity.db
   [2020-02-10 14:00:58.951 8283]    PARSE_ARP_CACHE: Active

   -----tail of pihole-FTL.log------
   [2020-02-10 16:56:27.080 3037] New forward server: 127.0.0.1 (-1/1024)
   [2020-02-10 16:56:27.080 3037] FATAL: Trying to access forward ID -1, but maximum is 1024
   [2020-02-10 16:56:27.080 3037]        found in findForwardID() (src/datastructure.c:54)
   [2020-02-10 16:56:27.080 3037] ERROR: Encountered serious memory error in findForwardID()
   [2020-02-10 16:56:27.087 3037] New forward server: 127.0.0.1 (-1/1024)
   [2020-02-10 16:56:27.088 3037] FATAL: Trying to access forward ID -1, but maximum is 1024
   [2020-02-10 16:56:27.088 3037]        found in findForwardID() (src/datastructure.c:54)
   [2020-02-10 16:56:27.088 3037] ERROR: Encountered serious memory error in findForwardID()
   [2020-02-10 16:56:27.091 3037] New forward server: 127.0.0.1 (-1/1024)
   [2020-02-10 16:56:27.091 3037] FATAL: Trying to access forward ID -1, but maximum is 1024
   [2020-02-10 16:56:27.091 3037]        found in findForwardID() (src/datastructure.c:54)
   [2020-02-10 16:56:27.091 3037] ERROR: Encountered serious memory error in findForwardID()
   [2020-02-10 16:56:27.095 3037] New forward server: 127.0.0.1 (-1/1024)
   [2020-02-10 16:56:27.095 3037] FATAL: Trying to access forward ID -1, but maximum is 1024
   [2020-02-10 16:56:27.095 3037]        found in findForwardID() (src/datastructure.c:54)
   [2020-02-10 16:56:27.095 3037] ERROR: Encountered serious memory error in findForwardID()
   [2020-02-10 16:56:27.099 3037] New forward server: 127.0.0.1 (-1/1024)
   [2020-02-10 16:56:27.099 3037] FATAL: Trying to access forward ID -1, but maximum is 1024
   [2020-02-10 16:56:27.099 3037]        found in findForwardID() (src/datastructure.c:54)
   [2020-02-10 16:56:27.099 3037] ERROR: Encountered serious memory error in findForwardID()
   [2020-02-10 16:56:27.957 3037] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   [2020-02-10 16:56:27.957 3037] ---------------------------->  FTL crashed!  <----------------------------
   [2020-02-10 16:56:27.957 3037] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   [2020-02-10 16:56:27.957 3037] Please report a bug at https://github.com/pi-hole/FTL/issues
   [2020-02-10 16:56:27.957 3037] and include in your report already the following details:
   [2020-02-10 16:56:27.957 3037] FTL has been running for 453 seconds
   [2020-02-10 16:56:27.957 3037] FTL branch: release/v5.0
   [2020-02-10 16:56:27.957 3037] FTL version: vDev-fb01e09
   [2020-02-10 16:56:27.957 3037] FTL commit: fb01e09
   [2020-02-10 16:56:27.957 3037] FTL date: 2020-02-10 10:20:26 +0100
   [2020-02-10 16:56:27.957 3037] FTL user: started as pihole, ended as pihole
   [2020-02-10 16:56:27.957 3037] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
   [2020-02-10 16:56:27.957 3037] Received signal: Segmentation fault
   [2020-02-10 16:56:27.957 3037]      at address: 0x338fe99c
   [2020-02-10 16:56:27.957 3037]      with code: SEGV_MAPERR (Address not mapped to object)

*** [ DIAGNOSING ]: contents of /dev/shm
-rw------- 1 pihole pihole 20480 Feb 10 16:48 /dev/shm/FTL-clients
-rw------- 1 pihole pihole 120 Feb 10 16:48 /dev/shm/FTL-counters
-rw------- 1 pihole pihole 12288 Feb 10 16:48 /dev/shm/FTL-dns-cache
-rw------- 1 pihole pihole 65536 Feb 10 16:48 /dev/shm/FTL-domains
-rw------- 1 pihole pihole 20480 Feb 10 16:48 /dev/shm/FTL-forwarded
-rw------- 1 pihole pihole 28 Feb 10 16:48 /dev/shm/FTL-lock
-rw------- 1 pihole pihole 53248 Feb 10 16:48 /dev/shm/FTL-overTime
-rw------- 1 pihole pihole 4096 Feb 10 16:48 /dev/shm/FTL-per-client-regex
-rw------- 1 pihole pihole 1376256 Feb 10 16:56 /dev/shm/FTL-queries
-rw------- 1 pihole pihole 12 Feb 10 16:48 /dev/shm/FTL-settings
-rw------- 1 pihole pihole 40960 Feb 10 16:49 /dev/shm/FTL-strings

*** [ DIAGNOSING ]: Locale
    LANG=

*** [ DIAGNOSING ]: Pi-hole log
-rw-r--r-- 1 pihole pihole 4299783 Feb 10 16:56 /var/log/pihole.log
   -----head of pihole.log------
   Feb 10 00:00:03 dnsmasq[17723]: validation result is INSECURE
   Feb 10 00:00:03 dnsmasq[17723]: reply 2a02:*PRIVACY* is NXDOMAIN
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 21.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: DHCP 192.168.178.21 is AVM1220E-0cbbd7.lan
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] b.c.0.b.c.a.e.f.f.f.4.1.e.0.a.c.0.0.e.5.7.9.2.a.0.7.0.8.2.0.a.2.ip6.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: forwarded b.c.0.b.c.a.e.f.f.f.4.1.e.0.a.c.0.0.e.5.7.9.2.a.0.7.0.8.2.0.a.2.ip6.arpa to 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: validation result is INSECURE
   Feb 10 00:00:03 dnsmasq[17723]: reply 2a02:*PRIVACY* is NXDOMAIN
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 1.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: config 192.168.178.1 is NXDOMAIN
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 126.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: DHCP 192.168.178.126 is EPSON3CEAAF.lan
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 22.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: config 192.168.178.22 is NXDOMAIN
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 180.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: DHCP 192.168.178.180 is COM-MID1.lan
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 4.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: /etc/pihole/custom.list 192.168.178.4 is raspi4.lan
   Feb 10 00:00:03 dnsmasq[17723]: query[PTR] 59.178.168.192.in-addr.arpa from 127.0.0.1
   Feb 10 00:00:03 dnsmasq[17723]: config 192.168.178.59 is NXDOMAIN


********************************************
********************************************
[✓] ** FINISHED DEBUGGING! **

    * The debug log can be uploaded to tricorder.pi-hole.net for sharing with developers only.
    * For more information, see: https://pi-hole.net/2016/11/07/crack-our-medical-tricorder-win-a-raspberry-pi-3/
    * If available, we'll use openssl to upload the log, otherwise it will fall back to netcat.
[i] Debug script running in automated mode
    * Using curl for transmission.
[✗]  There was an error uploading your debug log.
   * Please try again or contact the Pi-hole team for assistance.
   * A local copy of the debug log can be found at: /var/log/pihole_debug.log
1 Like

same problem here.
Pi-hole version is v4.3.2-397-g0a70bbd (Latest: v4.3.2)
AdminLTE version is v4.3.2-393-g843e46d (Latest: v4.3.2)
FTL version is vDev-fb01e09 (Latest: v4.3.1)

Crash here too now, using latest version from pihole -up

Same here: https://tricorder.pi-hole.net/jc9wpnmj1v

2 Likes

Get the same - see this in screen:

Thread 1197 "telnet-18" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb2ffe460 (LWP 23602)]
cmpdesc (a=0x32ffd9a0, b=0xb2ffd9a8) at src/api/api.c:53
53      src/api/api.c: No such file or directory.
(gdb) backtrace
#0  cmpdesc (a=0x32ffd9a0, b=0xb2ffd9a8) at src/api/api.c:53
#1  0xb6d6d280 in _quicksort (pbase=0x20, pbase@entry=0xb2ffd9a8, total_elems=total_elems@entry=4294967295, size=size@entry=8, cmp=cmp@entry=0x42a024 <cmpdesc>,
    arg=arg@entry=0x0) at qsort.c:122
#2  0xb6d6dae0 in __GI___qsort_r (b=b@entry=0xb2ffd9a8, n=4294967295, s=s@entry=8, cmp=0x42a024 <cmpdesc>, arg=arg@entry=0x0) at msort.c:216
#3  0xb6d6dbb8 in __GI_qsort (b=b@entry=0xb2ffd9a8, n=<optimized out>, s=s@entry=8, cmp=<optimized out>) at msort.c:308
#4  0x0042b314 in getForwardDestinations (client_message=client_message@entry=0xb3a1e5d0 ">forward-dest", sock=sock@entry=0xb2ffda28) at src/api/api.c:496
#5  0x00429480 in process_request (client_message=client_message@entry=0xb3a1e5d0 ">forward-dest", sock=sock@entry=0xb2ffda28) at src/api/request.c:65
#6  0x00428158 in telnet_connection_handler_thread (socket_desc=0x114e840) at src/api/socket.c:336
#7  0xb6e94494 in start_thread (arg=0xb2ffe460) at pthread_create.c:486
#8  0xb6e17578 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

FTL is chrashing here too´and flooding its error log this way:

[2020-02-11 08:18:53.483 353] New forward server: ::1 (-276/1024)
[2020-02-11 08:18:53.483 353] FATAL: Trying to access forward ID -276, but maximum is 1024
[2020-02-11 08:18:53.483 353] found in findForwardID() (src/datastructure.c:54)
[2020-02-11 08:18:53.483 353] ERROR: Encountered serious memory error in findForwardID()
[2020-02-11 08:18:53.484 353] New forward server: ::1 (-276/1024)
[2020-02-11 08:18:53.484 353] FATAL: Trying to access forward ID -276, but maximum is 1024
[2020-02-11 08:18:53.484 353] found in findForwardID() (src/datastructure.c:54)
[2020-02-11 08:18:53.484 353] ERROR: Encountered serious memory error in findForwardID()

Checking this now. I'm still unable to reproduce, but hopefully I will find what is causing this by looking again at the code. We've not touched the API in any way in Ensure each regex is imported only once by DL6ER · Pull Request #689 · pi-hole/FTL · GitHub so I'm not sure reverting this would fix it.

@Marble @jpgpi250 @Tntdruid @TschernoBill @scottydaman @pisome
Could anyone of you confirm if
pihole checkout ftl revert-689-ltaub
fixes the crashes permanently?

edit: Hold on, I know where the issue is, fix incoming....

edit 2: Please try

pihole checkout ftl fix/negative_forwardIDs

Just a moment pse. :slight_smile:

New forward server: ::1 (-276/1024)

would probably tell me, that there's something wrong with the DNS server config since Pihole should use the local running unbound. So I opened the WebIF, FTL instantly chrashed again, but I could chance DNS servers. After having FTL running, I changed DNS servers back to localhosts IPs an now the log of FTL looks like this:

[2020-02-11 09:17:03.806 4980] Received: Real-time signal 0 (34 -> 0)
[2020-02-11 09:17:03.819 4980] INFO: No regex whitelist entries found
[2020-02-11 09:17:03.828 4980] Compiled 0 whitelist and 218 blacklist regex filters in 14.6 msec
[2020-02-11 09:17:58.157 4980] Shutting down...
[2020-02-11 09:17:58.159 4980] Finished final database update
[2020-02-11 09:17:58.160 4980] ########## FTL terminated after 134774.9 ms! ##########
[2020-02-11 09:17:58.314 5279] Using log file /var/log/pihole-FTL.log
[2020-02-11 09:17:58.314 5279] ########## FTL started! ##########
[2020-02-11 09:17:58.314 5279] FTL branch: release/v5.0
[2020-02-11 09:17:58.314 5279] FTL version: vDev-fb01e09
[2020-02-11 09:17:58.314 5279] FTL commit: fb01e09
[2020-02-11 09:17:58.314 5279] FTL date: 2020-02-10 10:20:26 +0100
[2020-02-11 09:17:58.314 5279] FTL user: pihole
[2020-02-11 09:17:58.314 5279] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-11 09:17:58.314 5279] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2020-02-11 09:17:58.314 5279] SOCKET_LISTENING: only local
[2020-02-11 09:17:58.314 5279] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2020-02-11 09:17:58.314 5279] MAXDBDAYS: max age for stored queries is 30 days
[2020-02-11 09:17:58.314 5279] RESOLVE_IPV6: Resolve IPv6 addresses
[2020-02-11 09:17:58.314 5279] RESOLVE_IPV4: Resolve IPv4 addresses
[2020-02-11 09:17:58.314 5279] DBINTERVAL: saving to DB file every minute
[2020-02-11 09:17:58.314 5279] DBFILE: Using /etc/pihole/pihole-FTL.db
[2020-02-11 09:17:58.314 5279] MAXLOGAGE: Importing up to 24.0 hours of log data
[2020-02-11 09:17:58.314 5279] PRIVACYLEVEL: Set to 0
[2020-02-11 09:17:58.314 5279] IGNORE_LOCALHOST: Show queries from localhost
[2020-02-11 09:17:58.314 5279] BLOCKINGMODE: Null IPs for blocked domains
[2020-02-11 09:17:58.314 5279] ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2020-02-11 09:17:58.314 5279] DBIMPORT: Importing history from database
[2020-02-11 09:17:58.314 5279] PIDFILE: Using /var/run/pihole-FTL.pid
[2020-02-11 09:17:58.314 5279] PORTFILE: Using /var/run/pihole-FTL.port
[2020-02-11 09:17:58.314 5279] SOCKETFILE: Using /var/run/pihole/FTL.sock
[2020-02-11 09:17:58.314 5279] SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2020-02-11 09:17:58.315 5279] MACVENDORDB: Using /etc/pihole/macvendor.db
[2020-02-11 09:17:58.315 5279] GRAVITYDB: Using /etc/pihole/gravity.db
[2020-02-11 09:17:58.315 5279] PARSE_ARP_CACHE: Active
[2020-02-11 09:17:58.315 5279] REGEX_IGNORECASE: Disabled. Regex is case sensitive
[2020-02-11 09:17:58.315 5279] CNAME_DEEP_INSPECT: Active
[2020-02-11 09:17:58.315 5279] Finished config file parsing
[2020-02-11 09:17:58.316 5279] Database version is 5
[2020-02-11 09:17:58.316 5279] Database successfully initialized
[2020-02-11 09:17:58.317 5279] New forward server: ::1 (0/1024)
[2020-02-11 09:17:58.317 5279] New forward server: 127.0.0.1 (1/1024)
[2020-02-11 09:17:58.317 5279] New forward server: 192.168.201.254 (2/1024)
[2020-02-11 09:17:58.318 5279] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581330871
[2020-02-11 09:17:58.318 5279] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581330871
--------- THIS over 1943 lines! ---------------
[2020-02-11 09:17:58.402 5279] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408036
[2020-02-11 09:17:58.402 5279] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408036
[2020-02-11 09:17:58.402 5279] New forward server: 2606:4700:4700::1001 (3/1024)
[2020-02-11 09:17:58.402 5279] New forward server: 2620:fe::fe:10 (4/1024)
[2020-02-11 09:17:58.403 5279] New forward server: 2620:fe::10 (5/1024)
[2020-02-11 09:17:58.403 5279] Imported 5558 queries from the long-term database
[2020-02-11 09:17:58.403 5279] -> Total DNS queries: 5558
[2020-02-11 09:17:58.404 5279] -> Cached DNS queries: 2433
[2020-02-11 09:17:58.404 5279] -> Forwarded DNS queries: 1325
[2020-02-11 09:17:58.404 5279] -> Blocked DNS queries: 1796
[2020-02-11 09:17:58.404 5279] -> Unknown DNS queries: 4
[2020-02-11 09:17:58.404 5279] -> Unique domains: 443
[2020-02-11 09:17:58.404 5279] -> Unique clients: 10
[2020-02-11 09:17:58.404 5279] -> Known forward destinations: 6
[2020-02-11 09:17:58.404 5279] Successfully accessed setupVars.conf
[2020-02-11 09:17:58.410 5282] PID of FTL process: 5282
[2020-02-11 09:17:58.410 5282] Listening on port 4711 for incoming IPv4 telnet connections
[2020-02-11 09:17:58.410 5282] Listening on port 4711 for incoming IPv6 telnet connections
[2020-02-11 09:17:58.410 5282] Listening on Unix socket
[2020-02-11 09:17:58.411 5282] Reloading DNS cache
[2020-02-11 09:17:58.411 5282] Blocking status is enabled
[2020-02-11 09:17:58.423 5282] INFO: No regex whitelist entries found
[2020-02-11 09:17:58.433 5282] Compiled 0 whitelist and 218 blacklist regex filters in 16.1 msec

It seems, that its working, but the 1943 lines warning....

testing now

Yeah, I had a typo in there which was doing something really nasty. It's surprising this made it through our fairly complex automated testing environment, but this was a really subtle one.

ran pihole checkout ftl fix/negative_forwardIDs
errors in the log
stopped pihole-FTL
cleared all the logs
started pihole-FTL
result (removed a lot of FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408584 entries), to enable submit)

[2020-02-11 09:38:32.764 12707] Using log file /var/log/pihole-FTL.log
[2020-02-11 09:38:32.764 12707] ########## FTL started! ##########
[2020-02-11 09:38:32.764 12707] FTL branch: fix/negative_forwardIDs
[2020-02-11 09:38:32.764 12707] FTL version: vDev-409985b
[2020-02-11 09:38:32.764 12707] FTL commit: 409985b
[2020-02-11 09:38:32.764 12707] FTL date: 2020-02-11 09:22:30 +0100
[2020-02-11 09:38:32.764 12707] FTL user: pihole
[2020-02-11 09:38:32.764 12707] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-11 09:38:32.764 12707] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2020-02-11 09:38:32.764 12707]    SOCKET_LISTENING: only local
[2020-02-11 09:38:32.764 12707]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2020-02-11 09:38:32.764 12707]    MAXDBDAYS: max age for stored queries is 3 days
[2020-02-11 09:38:32.764 12707]    RESOLVE_IPV6: Resolve IPv6 addresses
[2020-02-11 09:38:32.764 12707]    RESOLVE_IPV4: Resolve IPv4 addresses
[2020-02-11 09:38:32.765 12707]    DBINTERVAL: saving to DB file every 120 seconds
[2020-02-11 09:38:32.765 12707]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2020-02-11 09:38:32.765 12707]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2020-02-11 09:38:32.765 12707]    PRIVACYLEVEL: Set to 0
[2020-02-11 09:38:32.765 12707]    IGNORE_LOCALHOST: Show queries from localhost
[2020-02-11 09:38:32.765 12707]    BLOCKINGMODE: Null IPs for blocked domains
[2020-02-11 09:38:32.765 12707]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2020-02-11 09:38:32.765 12707]    DBIMPORT: Importing history from database
[2020-02-11 09:38:32.765 12707]    PIDFILE: Using /var/run/pihole-FTL.pid
[2020-02-11 09:38:32.765 12707]    PORTFILE: Using /var/run/pihole-FTL.port
[2020-02-11 09:38:32.765 12707]    SOCKETFILE: Using /var/run/pihole/FTL.sock
[2020-02-11 09:38:32.765 12707]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2020-02-11 09:38:32.765 12707]    MACVENDORDB: Using /etc/pihole/macvendor.db
[2020-02-11 09:38:32.765 12707]    GRAVITYDB: Using /etc/pihole/gravity.db
[2020-02-11 09:38:32.765 12707]    PARSE_ARP_CACHE: Active
[2020-02-11 09:38:32.765 12707]    REGEX_IGNORECASE: Disabled. Regex is case sensitive
[2020-02-11 09:38:32.765 12707]    CNAME_DEEP_INSPECT: Active
[2020-02-11 09:38:32.765 12707] Finished config file parsing
[2020-02-11 09:38:32.767 12707] Database version is 5
[2020-02-11 09:38:32.767 12707] Database successfully initialized
[2020-02-11 09:38:32.768 12707] New forward server: fdaa:bbcc:ddee:2::5552 (0/1024)
[2020-02-11 09:38:32.774 12707] Resizing "/FTL-strings" from 4096 to 8192
[2020-02-11 09:38:32.812 12707] New forward server: 127.10.10.2 (1/1024)
[2020-02-11 09:38:32.822 12707] Resizing "/FTL-queries" from 196608 to 393216
[2020-02-11 09:38:32.849 12707] Resizing "/FTL-strings" from 8192 to 12288
[2020-02-11 09:38:32.853 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581405234
status QUERY_FORWARDED, 1581405235
[2020-02-11 09:38:32.854 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581405236

[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408584
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408584
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408584
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408584
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408649
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408651
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408678
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408678
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408682
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408682
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408683
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408690
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408697
[2020-02-11 09:38:32.872 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408721
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408900
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408960
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408987
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408990
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581408990
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409027
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409093
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409117
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409126
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409190
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409195
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409313
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409313
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409334
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409353
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409353
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409353
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409353
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409354
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409354
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409408
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409413
[2020-02-11 09:38:32.873 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409419
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409425
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409428
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409429
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409429
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409430
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409431
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409434
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409435
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409439
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409440
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409449
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409450
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409454
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409455
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409459
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409460
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409558
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409558
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409579
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409579
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409608
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409608
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409616
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409616
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409618
[2020-02-11 09:38:32.874 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409618
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409643
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409643
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409657
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409868
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409868
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409869
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409869
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409884
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409885
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409886
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409886
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409889
[2020-02-11 09:38:32.875 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581409889
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410062
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410099
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410099
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410110
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410173
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410173
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410173
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410173
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] FTL_db warn: FORWARD should not be NULL with status QUERY_FORWARDED, 1581410174
[2020-02-11 09:38:32.876 12707] Imported 6803 queries from the long-term database
[2020-02-11 09:38:32.877 12707]  -> Total DNS queries: 6803
[2020-02-11 09:38:32.877 12707]  -> Cached DNS queries: 142
[2020-02-11 09:38:32.877 12707]  -> Forwarded DNS queries: 5210
[2020-02-11 09:38:32.877 12707]  -> Blocked DNS queries: 1451
[2020-02-11 09:38:32.877 12707]  -> Unknown DNS queries: 0
[2020-02-11 09:38:32.877 12707]  -> Unique domains: 375
[2020-02-11 09:38:32.877 12707]  -> Unique clients: 5
[2020-02-11 09:38:32.877 12707]  -> Known forward destinations: 2
[2020-02-11 09:38:32.877 12707] Successfully accessed setupVars.conf
[2020-02-11 09:38:32.881 12709] PID of FTL process: 12709
[2020-02-11 09:38:32.881 12709] Listening on port 4711 for incoming IPv4 telnet connections
[2020-02-11 09:38:32.882 12709] Listening on port 4711 for incoming IPv6 telnet connections
[2020-02-11 09:38:32.882 12709] Listening on Unix socket
[2020-02-11 09:38:32.883 12709] Reloading DNS cache
[2020-02-11 09:38:32.883 12709] Blocking status is enabled
[2020-02-11 09:38:32.893 12709] INFO: No regex whitelist entries found
[2020-02-11 09:38:32.897 12709] Compiled 0 whitelist and 19 blacklist regex filters in 7.3 msec

This is expected, the forwarded data was broken before and hence the records in the database are now off. This will "fix itself" after the history window of 24 hours has passed.

stopped pihole-FTL
removed logs and pihole-FTL.db
started pihole-FTL
result (no errors):

[2020-02-11 09:48:51.942 14962] Using log file /var/log/pihole-FTL.log
[2020-02-11 09:48:51.942 14962] ########## FTL started! ##########
[2020-02-11 09:48:51.942 14962] FTL branch: fix/negative_forwardIDs
[2020-02-11 09:48:51.942 14962] FTL version: vDev-409985b
[2020-02-11 09:48:51.942 14962] FTL commit: 409985b
[2020-02-11 09:48:51.942 14962] FTL date: 2020-02-11 09:22:30 +0100
[2020-02-11 09:48:51.942 14962] FTL user: pihole
[2020-02-11 09:48:51.942 14962] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-11 09:48:51.942 14962] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2020-02-11 09:48:51.942 14962]    SOCKET_LISTENING: only local
[2020-02-11 09:48:51.942 14962]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2020-02-11 09:48:51.942 14962]    MAXDBDAYS: max age for stored queries is 3 days
[2020-02-11 09:48:51.942 14962]    RESOLVE_IPV6: Resolve IPv6 addresses
[2020-02-11 09:48:51.943 14962]    RESOLVE_IPV4: Resolve IPv4 addresses
[2020-02-11 09:48:51.943 14962]    DBINTERVAL: saving to DB file every 120 seconds
[2020-02-11 09:48:51.943 14962]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2020-02-11 09:48:51.943 14962]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2020-02-11 09:48:51.943 14962]    PRIVACYLEVEL: Set to 0
[2020-02-11 09:48:51.943 14962]    IGNORE_LOCALHOST: Show queries from localhost
[2020-02-11 09:48:51.943 14962]    BLOCKINGMODE: Null IPs for blocked domains
[2020-02-11 09:48:51.943 14962]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2020-02-11 09:48:51.943 14962]    DBIMPORT: Importing history from database
[2020-02-11 09:48:51.943 14962]    PIDFILE: Using /var/run/pihole-FTL.pid
[2020-02-11 09:48:51.943 14962]    PORTFILE: Using /var/run/pihole-FTL.port
[2020-02-11 09:48:51.943 14962]    SOCKETFILE: Using /var/run/pihole/FTL.sock
[2020-02-11 09:48:51.943 14962]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2020-02-11 09:48:51.943 14962]    MACVENDORDB: Using /etc/pihole/macvendor.db
[2020-02-11 09:48:51.943 14962]    GRAVITYDB: Using /etc/pihole/gravity.db
[2020-02-11 09:48:51.943 14962]    PARSE_ARP_CACHE: Active
[2020-02-11 09:48:51.943 14962]    REGEX_IGNORECASE: Disabled. Regex is case sensitive
[2020-02-11 09:48:51.943 14962]    CNAME_DEEP_INSPECT: Active
[2020-02-11 09:48:51.944 14962] Finished config file parsing
[2020-02-11 09:48:51.944 14962] No database file found, creating new (empty) database
[2020-02-11 09:48:52.136 14962] Database version is 3
[2020-02-11 09:48:52.136 14962] Updating long-term database to version 4
[2020-02-11 09:48:52.153 14962] Updating long-term database to version 5
[2020-02-11 09:48:52.211 14962] Database successfully initialized
[2020-02-11 09:48:52.212 14962] Imported 0 queries from the long-term database
[2020-02-11 09:48:52.212 14962]  -> Total DNS queries: 0
[2020-02-11 09:48:52.213 14962]  -> Cached DNS queries: 0
[2020-02-11 09:48:52.213 14962]  -> Forwarded DNS queries: 0
[2020-02-11 09:48:52.213 14962]  -> Blocked DNS queries: 0
[2020-02-11 09:48:52.213 14962]  -> Unknown DNS queries: 0
[2020-02-11 09:48:52.213 14962]  -> Unique domains: 0
[2020-02-11 09:48:52.213 14962]  -> Unique clients: 0
[2020-02-11 09:48:52.213 14962]  -> Known forward destinations: 0
[2020-02-11 09:48:52.213 14962] Successfully accessed setupVars.conf
[2020-02-11 09:48:52.220 14964] PID of FTL process: 14964
[2020-02-11 09:48:52.221 14964] Listening on port 4711 for incoming IPv4 telnet connections
[2020-02-11 09:48:52.221 14964] Listening on port 4711 for incoming IPv6 telnet connections
[2020-02-11 09:48:52.222 14964] Listening on Unix socket
[2020-02-11 09:48:52.224 14964] Reloading DNS cache
[2020-02-11 09:48:52.224 14964] Blocking status is enabled
[2020-02-11 09:48:52.236 14964] INFO: No regex whitelist entries found
[2020-02-11 09:48:52.236 14964] Compiled 0 whitelist and 19 blacklist regex filters in 5.8 msec

Do we need to keep this branch running, OR are you sure you identified the problem and will merge the fix into beta5?

Yes, however, merging into beta5 still requires review and approval, I'll report back.

I did this. And it seems it was helpful. The FTL-log after a reboot looks similar to yours an I hope FTL keeps running for more than 5 minutes now. :slight_smile:
We'll see.

Seems fine to me too! Here’s the Debuglog with the latest fix: https://tricorder.pi-hole.net/hmz6e00fh2

Thanks!

One hour later... FTL chrashed again.

[2020-02-11 10:50:25.022 2766] New forward server: ::1 (-1/1024)
[2020-02-11 10:50:25.022 2766] FATAL: Trying to access forward ID -1, but maximum is 1024
[2020-02-11 10:50:25.022 2766] found in findForwardID() (src/datastructure.c:54)
[2020-02-11 10:50:25.022 2766] ERROR: Encountered serious memory error in findForwardID()
[2020-02-11 10:50:44.163 2766] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-11 10:50:44.163 2766] ----------------------------> FTL crashed! <----------------------------
[2020-02-11 10:50:44.163 2766] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-02-11 10:50:44.163 2766] Please report a bug at Issues · pi-hole/FTL · GitHub
[2020-02-11 10:50:44.163 2766] and include in your report already the following details:
[2020-02-11 10:50:44.163 2766] FTL has been running for 1552 seconds
[2020-02-11 10:50:44.163 2766] FTL branch: release/v5.0
[2020-02-11 10:50:44.163 2766] FTL version: vDev-fb01e09
[2020-02-11 10:50:44.163 2766] FTL commit: fb01e09
[2020-02-11 10:50:44.163 2766] FTL date: 2020-02-10 10:20:26 +0100
[2020-02-11 10:50:44.163 2766] FTL user: started as pihole, ended as pihole
[2020-02-11 10:50:44.163 2766] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-02-11 10:50:44.163 2766] Received signal: Segmentation fault
[2020-02-11 10:50:44.163 2766] at address: 0x336fe99c
[2020-02-11 10:50:44.163 2766] with code: SEGV_MAPERR (Address not mapped to object)

And its the

New forward server: ::1 (-1/1024)

again.

This is a binary from yesterday,

and not the one on my proposed fix branch. Please run