Unbound frequent restarts

Maybe.
Note that some networking tools would ignore those settings, relying on their own configuration options instead - I believe NetworkManager may fall into that category.


If we cannot find this on the receiving end, let's try and capture it at the sender, using auditd.

EDIT: I've successfully tested below steps on Raspbian Buster, but couldn't get them to work on Armbian (the audit log never populated).
It may only be worth the hassle if you would run Raspberry Pi OS as well.

Since we don't know the sender, we have to capture traffic of all processes, so this may incur a performance hit for all of your system, and the respective log file may grow large quickly. And we still need strace to find the pid.

Don't forget to disable it once finished.

We start by installing auditd:

sudo apt install auditd audispd-plugins

Enable auditing:

sudo auditd -s enable

Find your system CPU architecture, e.g.:

~ $ uname -m
armv7l

Use that arch when creating a rule to capture system calls to kill -HUP:

sudo auditctl -a exit,always -F arch=armv7l -S kill -F a1=1

EDIT: In an attempt to limit audit log growth, I tried to be as restrictive as possible here, by assuming that unbound is indeed HUPed by an unknown process. I think that's a safe assumption, as the observed behaviour is documented for HUP reception in unbound. Still, if it proves too restrictive and you can't find a match, you may drop the trailing -F a1=1.

Verify that rule got created, e.g.:

~ $ sudo auditctl -l
-a always,exit -F arch=b32 -S kill -F a1=0x1

You should see an audit log at /var/log/audit/audit.log
Check that audit logging is working:

sudo auditctl -m 'verify its working'

Verify that statement made it into your audit log:

sudo grep -n 'verify' /var/log/audit/audit.log

If everything works, have your strace running also and wait for unbound to restart.

Once that happens, extract the pid of the process that HUPed unbound from your strace log as before, and use that to determine the offending process:

sudo grep -n -B1 '=29812 ' /var/log/audit/audit.log

In the above, replace 29812 with the pid from your strace (keep the leading = and trailing space).

That should give you a few lines like:

1265-type=USER_START msg=audit(1622277865.428:1119): pid=29811 uid=0 auid=1000 ses=1522 msg='op=PAM:session_open grantors=pam_permit,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
1266:type=SYSCALL msg=audit(1622277865.448:1120): arch=40000028 syscall=37 per=800000 success=yes exit=0 a0=1f6 a1=1 a2=42adc00 a3=42adc00 items=0 ppid=29811 pid=29812 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=1522 comm="kill" exe="/bin/kill" key=(null)

where exe= should point to the binary (at the end of line 1266 in the example above).
Let's hope that isn't just a shell /bin/kill spawned by yet another unknown process (in the example I used, it is of course such a command that I used to trigger those log lines).

Once done, don't forget to disable auditing:

sudo auditd -s disable
sudo service auditd stop
sudo systemctl disable auditd
4 Likes