Embedded web server randomly stops responding while DNS continues to work (curl to localhost hangs)

Help
1

Expected Behaviour:

The integrated Pi-hole web server should always respond to HTTP requests.

The web interface should remain reachable on port 80 for weeks without requiring a restart of the FTL service.

System:

  • Raspberry Pi 3B
  • Raspberry Pi OS (64-bit)
  • Linux 6.18.29+rpt-rpi-v8
  • Pi-hole Core v6.4.2
  • Web v6.5
  • FTL v6.6.2

No reverse proxy, nginx, Apache or Docker is used.
This is a standard Pi-hole installation using the integrated web server.

Actual Behaviour:

After running normally for several days (sometimes weeks), the integrated web server stops responding.

Symptoms:

  • DNS resolution continues to work perfectly.
  • Clients continue using Pi-hole normally.
  • SSH access works.
  • FTL service is still active.
  • Port 80 is still listening.
  • Browser access to /admin times out.

The problem can also be reproduced locally on the Pi itself.

Running

curl -v http://127.0.0.1/

only prints

* Trying 127.0.0.1:80...

and then hangs forever.

The same happens with

curl -v http://127.0.0.1/admin/
curl -v http://127.0.0.1/api/info

No HTTP response is ever returned.

At the same time:

systemctl status pihole-FTL

shows the service is running normally.

ss -tlnp

shows that pihole-FTL is listening on port 80.

Memory usage and CPU load remain completely normal.

Immediately after

sudo systemctl restart pihole-FTL

the web interface starts working again and

curl -v http://127.0.0.1/admin/

returns the expected HTTP 302 redirect.

This issue has now occurred multiple times.

Debug Token:

iw675ctQ

Is there any additional logging or debugging I could enable to help identify why the embedded HTTP server stops responding while the DNS functionality continues working?

2

Yeah a whole bunch (double TAB for Bash completion):

$ sudo pihole-FTL --config debug.
debug.aliasclients  debug.extra         debug.regex
debug.all           debug.flags         debug.reserved
debug.api           debug.gc            debug.resolver
debug.arp           debug.helper        debug.shmem
debug.caps          debug.inotify       debug.status
debug.clients       debug.locks         debug.timing
debug.config        debug.netlink       debug.tls
debug.database      debug.networking    debug.vectors
debug.dnssec        debug.ntp           debug.webserver
debug.edns0         debug.overtime
debug.events        debug.queries

Eg:

$ sudo pihole-FTL --config debug.webserver
false

$ sudo pihole-FTL --config debug.webserver true
true

That results in verbose logging to:

$ ls -1 /var/log/pihole/webserver.log*
/var/log/pihole/webserver.log
/var/log/pihole/webserver.log.1
/var/log/pihole/webserver.log.2.gz
/var/log/pihole/webserver.log.3.gz

Can use zcat or zless to view those .gz archived files.

Those debugging features are sublime!

3

Thanks! I have enabled the webserver debug logging as suggested and restarted FTL.

Since the issue only occurs after several days (or maybe weeks) of uptime, I’ll have to wait until it happens again. As soon as it does, I’ll collect the requested logs before restarting FTL and post them here.

4

The issue has now occurred again, this time on the current versions:

  • Pi-hole Core v6.4.3
  • Web v6.6
  • FTL v6.7

DNS resolution continued to work normally and SSH access was fully responsive. pihole-FTL remained active.

A local request timed out after approximately 135 seconds:

curl -v http://127.0.0.1/admin/
* Trying 127.0.0.1:80...
* connect to 127.0.0.1 port 80 failed: Connection timed out

ss showed that both IPv4 accept queues were full:

LISTEN 201 200 0.0.0.0:80
LISTEN 201 200 0.0.0.0:443
LISTEN   0 200 [::]:80
LISTEN   0 200 [::]:443

The file descriptors are:

fd 37 = IPv4 port 80
fd 38 = IPv4 port 443
fd 39 = IPv6 port 80
fd 40 = IPv6 port 443

The kernel counters showed:

TcpExtListenOverflows 9323
TcpExtListenDrops     9323

strace on the civetweb-master thread showed a very fast loop like:

ppoll([{fd=37, events=POLLIN},
       {fd=38, events=POLLIN},
       {fd=39, events=POLLIN},
       {fd=40, events=POLLIN},
       {fd=36, events=POLLIN}], ...)
    = 2 ([{fd=37, revents=POLLIN},
          {fd=38, revents=POLLIN}], ...)

This repeated continuously, but no accept() or accept4() call was visible. The FTL process was using approximately 58.6% CPU.

It therefore appears that the CivetWeb master thread sees pending IPv4 connections but no longer accepts them. The kernel accept queues fill up and subsequently drop new SYN packets, while DNS continues operating.

This appears very similar to the older topic titled “civetweb-master at 100% with unresponsive GUI”.

New debug token: EhiX6lWi

I can also provide a longer strace output if needed.

5

is this similar to what I experience?

I've found that it's dying at 11 past the hours, but also that the pihole server can't be found from other machines, like it's lost it's won dns entry.

it gets restored at 00 of the hour. And I've found that if I run a dig command on the pihole server looking at itself it actually resolves the issue - this is the work around I've put in currently

6

Thanks for sharing this. There is some overlap, as the web interface becomes unavailable while DNS queries are still answered. However, the underlying symptoms currently seem different.

In my case, the issue occurs irregularly after several days. It also affects direct access by IP and even 127.0.0.1:80. During the failure, the IPv4 accept queues for ports 80 and 443 are full (201/200), the kernel reports thousands of ListenOverflows and ListenDrops, and the civetweb-master thread loops continuously in ppoll() without accepting the pending connections. Restarting pihole-FTL immediately restores access.

Your hourly timing, the failed resolution of the Pi-hole’s own hostname, and the fact that a local dig restores it suggest that your issue may instead be related to local DNS resolution or a scheduled process.

If it happens again, it would be interesting to compare these outputs:

curl -v http://127.0.0.1/admin/
sudo ss -ltnp '( sport = :80 or sport = :443 )'
sudo nstat -az | grep -E 'ListenOverflows|ListenDrops'

If your accept queues are also full, the two issues might indeed be related.

7

I ended up rebuilding a new lsc on trixie and restoring the teleporter backup.

So never got to the bottom of it.

I did have some IO issues that I got to the bottom of on proxmox where LXC containers had their underlying mapped lvm datastores at high 90's for freespace (even though internally the lxc said it was at 50% used).

So I wonder if that's what was killing my pihole instance.

8

A short update, since this issue is still being monitored:

The problem occurred once more on July 27. DNS resolution was still working normally, but this time the web interface was unavailable and SSH connections were immediately closed by the Pi, so I could not collect the same live diagnostics as during the previous occurrence.

I had to power-cycle the Raspberry Pi. Unfortunately, I then discovered that the systemd journal was still configured as volatile, so the journal from the affected boot had not survived the restart.

I have now explicitly changed journald to:

Storage=persistent

and verified after a controlled reboot that previous boots are retained and accessible with:

journalctl --list-boots
journalctl -b -1

The Pi-hole has been running normally again for about 11 days now. I am leaving the webserver debugging enabled and will collect the persistent journal from the affected boot if/when the issue occurs again.

So at this point I am still waiting for the next occurrence, but the system is now prepared for a proper post-mortem even if SSH becomes unavailable again.