Unable to reach web after OS upgrade (default lighttpd /w mod_openssl + debian bullseye)

[FoUStep]

Expected Behaviour:

Upgrade my VMware VM from Debian Buster to Bullseye, was running the latest version of Pi-Hole. Changed the sources in apt from buster to bullseye (11.2). Update & dist-upgrade, reboot. All works except my https using lighttpd configured web interface, it worked fine before the OS upgrade and it always worked after the upgrading Pi-Hole components/

Actual Behaviour:

The web interface (v5.8) does not work after updating the OS. Lighttpd does start without any journal errors.

• eth0 is the network interface, IPv4 did not change, IPv6 is disabled
• all services work except the web interface
• *:443 lighttpd (IPv4)
• no firewall

Packages:

- lighttpd-mod-openssl                 1.4.59-1                       amd64        TLS support using OpenSSL module for lighttpd
- openssl                              1.1.1k-1+deb11u1

Info:

*** [ DIAGNOSING ]: Pi-hole processes
[✓] lighttpd daemon is active
[✓] pihole-FTL daemon is active

/var/log/lighttpd/error.log

2021-12-18 18:28:33: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-12-18 18:28:33: server.c.1513) server started (lighttpd/1.4.59)
2021-12-18 18:28:33: gw_backend.c.475) unlink /run/lighttpd/php.socket-0 after connect failed: Connection refused

my /etc/lighttpd/external.conf:

server.modules += (
   "mod_openssl"
)

# Ensure the Pi-hole Block Page knows that this is not a blocked domain
setenv.add-environment = ("fqdn" => "true")

$HTTP["host"] == "my.local.ip" {
  $SERVER["socket"] == ":443" {
    ssl.engine = "enable"
    ssl.pemfile = "/etc/lighttpd/ssl/combined.pem"
    ssl.honor-cipher-order = "enable"
    ssl.cipher-list = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
  }

  # Redirect HTTP to HTTPS
  $HTTP["scheme"] == "http" {
    $HTTP["host"] =~ ".*" {
      url.redirect = (".*" => "https://%0$0")
    }
  }
}

/var/log/messages:

Dec 18 18:52:42 infra kernel: [ 3036.931015] lighttpd[10098]: segfault at 8 ip 00007f68c9563a9a sp 00007ffe51d6a700 error 4 in mod_openssl.so[7f68c9561000+6000]
Dec 18 18:52:42 infra kernel: [ 3036.931028] Code: 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 e8 e8 df ff ff 66 83 78 1c 04 0f 84 dd 00 00 00 4c 8b 60 20 48 89 c3 <49> 8b 74 24 08 48 85 f6 0f 84 9c e1 ff ff 49 83 3c 24 00 0f 84 91

I tried several things:

• Changed OpenSSL(.cnf) with a minimum protocol version to TLSv1.0
• Changed external.conf and added ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1") below ssl.cipher-list
• Rebooted several times after changes
• All packages are updated and upgraded. Apt auto removed old packages.
• Tried to google the issues above to no avail unfortunately, there is something wrong or missing in the SSL configuration, could be that protocol minimum has been changed and the configuration can't handle it.
• Reinstalled & reconfigured Pi-Hole.

Does anyone have any idea what else to try to solve this?

[DanSchaper]

Looks like a segfault, which often means system corruption or file corruption.

This isn't a Pi-hole issue, it's a system level issue.

[FoUStep]

Thank you for your time Dan. Is there any way to check this? I'm also running a UniFi controller since I first installed this years ago on the same host without any issues, that one is running without any issues whatsoever. Reinstalling packages is not a problem, I should be able to force installing whatever is necessary. Did not have any issues upgrading from Debian 9 to Debian 10 back in the day and even waited a while since the release of Bullseye before upgrading.

[DanSchaper]

@FoUStep

mod_openssl.so

That's the file that is segfaulting, you might be able to replace that library with the current one, or try reinstalling that package, I'm not sure which one it would be though.

[FoUStep]

Moved /usr/lib/lighttpd/mod_openssl.so to my personal folder. Reinstalled it through:

sudo apt-get reinstall lighttpd-mod-openssl

It reinstalled and restarted lighttpd but unfortunately it did not solve the issue.

[FoUStep]

So I did some hardcore digging.

The issue seems to have been this in the error.log:
gw_backend.c.475) unlink /run/lighttpd/php.socket-0 after connect failed: Connection refused

Very hard to find but I came across this.

Apparently you are not allow to nest "$SERVER["socket"]", and so I removed this from my external.conf:
$HTTP["host"] == "my.local.ip" { }

I had copy-paste the external.conf from somewhere, I did not figure that out myself.

After removing that line (and thus unnesting the $SERVER socket) and restarting lighttpd, it all seems to work again. Tough find, hopefully useful for someone who has the same issue in the future.