Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Connection prematurely closed by remote server)

Help
41

One new occurence appears on Pihole web interface but unable to verify the number of incidents in the FTL.log (unsure what triggers this log to purge itself)

UPDATE, after a helpful suggestion by @Ladrien on a different thread:

Two new occurences for me:

2025-03-12 21:15:43.713 GMT [220081/F127218] WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Resource temporarily unavailable)
2025-03-12 21:15:43.969 GMT [220082/F127218] WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Resource temporarily unavailable)

42

I have the below entries in unbound.conf.d/pi-hole.conf
Currently I haven't seen any errors.

server:
    # This setting should increase the number of TCP connections that stop the pi-hole errors
    incoming-num-tcp: 50
    tcp-idle-timeout: 1024
    outgoing-range: 8192
    num-queries-per-thread: 4096

    # If no logfile is specified, syslog is used
    logfile: "/var/log/unbound/unbound.log"
    log-time-ascii: yes
    log-queries: yes
    log-replies: yes
    verbosity: 0
43

In my setup, the /etc/unbound/unbound.conf has a single include statement
include-toplevel: "/etc/unbound/unbound.conf.d/*.conf", whereas the configuration is in /etc/unbound/unbound.conf.d/pi-hole.conf. I assume your suggested modifications should be on this conf file on my setup; correct?

1 Like
44

Oops :slight_smile: post updated
Thanks for the catch

/etc/unbound/unbound.conf.d/pi-hole.conf:server:
/etc/unbound/unbound.conf.d/pi-hole.conf:    incoming-num-tcp: 50
/etc/unbound/unbound.conf.d/pi-hole.conf:    tcp-idle-timeout: 1024
/etc/unbound/unbound.conf.d/pi-hole.conf:    outgoing-range: 8192
/etc/unbound/unbound.conf.d/pi-hole.conf:    num-queries-per-thread: 4096
/etc/unbound/unbound.conf.d/pi-hole.conf:    logfile: "/var/log/unbound/unbound.log"
/etc/unbound/unbound.conf.d/pi-hole.conf:    log-time-ascii: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    log-queries: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    log-replies: yes
/etc/unbound/unbound.conf.d/pi-hole.conf:    verbosity: 0
/
45

It did not go well with my setup. I have had multiple intermittent occurences of the error since making these modifications to the /etc/unbound/unbound.conf.d/pi-hole.conf file. I reverted "incoming-num-tcp" to 1024 and monitoring.

47

It is strange...
I got two instances of pi-hole, one on Wired LAN and the other on Wireless LAN, on the wireless LAN I almost never see the issue, while on the wired LAN I see the error, just since the change I did I haven't seen one instance on either of the two pi-hole instances.
Initial I had the idea it was browser related, but not...

Check https://github.com/NLnetLabs/unbound/issues/1237#issuecomment-2658989107 where we opened the issue with the unbound team.

48

This is the top lines of my /etc/unbound/unbound.conf.d/pi-hole.conf:

Screenshot_5
Screenshot_5853×293 46 KB

I have had 3 occurences since my last modification to the conf file

2025-03-14 16:10:27.267 EET [2295/F931] WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Connection prematurely closed by remote server)
<...>
2025-03-14 19:17:58.497 EET [3259/F931] WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Connection prematurely closed by remote server)
2025-03-14 19:17:58.503 EET [3260/F931] WARNING: Connection error (127.0.0.1#5335): TCP connection failed while receiving payload length from upstream (Connection prematurely closed by remote server)

I am using a single PiHole instance with Unbound, on a Pi4B/4GB and a wired connection.

49

Please post this in the unbound forum, link in my previous post.

1 Like
50

Since the pihole v5->v6 upgrade, I'm seeing the same errors a couple of times a day

Screenshot 2025-03-26 222106
Screenshot 2025-03-26 2221061920×1080 231 KB

51

3 posts were split to a new topic: Wrong port in log message

55

I have been curious about this since I have been getting this warning
about once a week for well over a year, under both v5 and v6.
So, from my experience, the upgrade was not the issue.
Hopefully, someone will identify the issue, or it will be corrected in a subsequent PiHole update

56

I am receiving multiple persistent occurences of this error today

image
image1302×1674 401 KB

57

The issue discussed here seems to be not an unbound issue. That's why changing unbound config doesn't help. The key statement from the unbound GitHub open issue related to this one is:

dnsmasq queries Unbound over TCP (not sure why, probably because dnsmasq was queried over TCP as well),

Link: failed to send TCP(read_write) packet (Connection prematurely closed by remote server) · Issue #1237 · NLnetLabs/unbound · GitHub
Is there a reason why dnsmasq started to query unbound over TCP more often in pi-hole v6? This warning/error didn't occur in v5.

58

Pi-hole v5 didn't log Connection prematurely closed by remote server messages, they have been added with v6, see Periodic CONNECTION_ERRORs in Diagnostics - Failed to send TCP (read_write) to 127.0.0.1#5335 - #14 by DL6ER.

That's not the key statement.
TCP retries are perfectly normal.
The DNS protocol encourages clients to use UDP by default, but if a DNS reply does exceed UDP packet length, a client may resend the DNS request via TCP, which supports multi packet transport.

The key statement is this:

The summary is that this is not an Unbound issue. The "issue" is extra harmless logging on pihole v6.

With regards to the message itself, I have a differentiated view:

No, Connection prematurely closed by remote server definitely is caused by unbound throwing away TCP connections.

The linked GitHub issue has identified two potential causes for this:
a. unbound exceeds its timeout threshold when walking the recursion chain, which is more likely to happen for misconfigured domains, and unbound then terminates the connection.
b. a new TCP request from a client exceeds unbound's TCP connection pool, so unbound terminates a TCP connection.

The former could only be addressed by adjusting configuration on the authoritative DNS servers by the maintainers of the misconfigured domain.

Increasing unbound's incoming-num-tcp over its default of 10 would affect the latter. Note that unbound maintainers considered already a value of 100 as excessive (requiring more memory), and increasing this value beyond 150 (Pi-hole's internal limit on concurrent connections) would not be likely to result in further improvements.

Depending on the cause for termination of TCP connection, it could still be beneficial to adjust unbound, but even then, the message could still be triggered under heavy load or when querying a misconfigured domain.

While the client whose DNS query prompted that message will have to repeat its query, this doesn't impact DNS resolution in general - that is to say it wouldn't be unusual to observe the message occasionally.

4 Likes
59

I got this error for the first time today. Updated to V6 last week. I noticed this only happens when I go to a certain streaming site, but it totally nukes my entire network for 60 seconds. It feels tied to the rate limit feature.

60

Similarly observations. Throws random errors. Sometimes it blocks connections for some time (a minute sometimes two). Everything from Pihole6. I even changed the raspberry installations to a stronger x86 machine. It did nothing. Pihole Nr6 looks broken. I have no idea to remove these persistent mistakes.

61

My latest observation: I have modified the incoming-num-tcp to a sensible number (96) in the /etc/unbound/unbound.conf.d/pi-hole.conf, as per @piholeftw suggestion and have not experienced any errors since the last reboot, 72hrs ago.

62

It seems to help, but sometimes a connection error can appear again.
However, I think that the 6th version of Pihole is defective, because even Web Interface sometimes hangs.

I hope that the creators are working on improving several problems.

63

Errors start popping up once again, however it has been suggested that these appear on v6 because the granularity of what is being resported has been modified on v6 and the system still works fine.