Error in /var/log/pihole_updateGravity.log

fresh install of pihole v5.1.1, branch master

changing line 122 from

svc="service ${resolver} restart"

into

svc="/usr/sbin/service ${resolver} restart"

solves the problem

Does this imply that none of the deployed pihole v5.1.1 did restart pihole-FTL this night, the new gravity download isn't active?

My cron looks like

41 4   * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log

Which calls

and at the end restarts pihole

So I'm not sure why line 129 is involved here at all, as the relevant line for reload is

I can only report what I see.

My conclusion (may be wrong): The reason why the service command fails: cron doesn't use the $PATH, that would be used when executing pihole -g (user is pi). I've read the default path is hardcoded in the cron binary, and doesn't include what we need.

Adding the full path to the script solves the problem, although this may not be the best solution.

I've been looking at methods to ensure cron also knows about the $PATH variable, a regular user uses. One of the methods found, refers to to /etc/crontab (the system-wide crontab). This file has an additional shell and path command, that includes /usr/sbin

I would never have caught this error, if I didn't split the cron command

59 1    * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log

into

59 1   * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log
9 2   * * 7   root    PATH="$PATH:/bin/cat/" cat /var/log/pihole_updateGravity.log

The reason I have to do this: I've configured my system to send mails, when using cron, mails are sent by default, unless >/dev/null 2>&1 is added.
Because pihole updateGravity is restarting (or should be) pihole-FTL, the system cannot resolve the mailserver domain, defined in my MSMTP config file, the mail is never send (pihole-FTL doesn't resolve yet, not fully ready). By delaying the cat command (the output is send as a mail) this problem is solved.

Anyhow, the problem is temporarily solved, by changing the script, just wondering how many other users aren't even aware they have a problem...

I do not have this problem (Latest master). The contents of the log file are the same as the output of pihole -g.

Edit - problem not seen on V5.1, V5.0, V4.x, and two different dev versions currently running on different devices (all running various flavors of Buster).

Please post the token generated by

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log


[✓] Your debug token is: https://tricorder.pi-hole.net/s1eccmm8fc


apparently, pihole-FTL crashed at 2020-07-19 08:58 (didn't notice that, no user inconvenience)
but this is NOT when the cron job runs.

there is a group allowAqueriesOnly, but only clients are assigned, no regexes (prepared for regex engine improvements)

Debug log looks normal except for the crash.

Have you checked the function of the service command with Pi-hole?

sudo service pihole-FTL restart

sudo service pihole-FTL status

sudo service pihole-FTL status
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: active (exited) since Sun 2020-07-19 09:44:22 CEST; 5h 33min ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0 (limit: 2200)
   Memory: 0B
   CGroup: /system.slice/pihole-FTL.service

Jul 19 09:44:21 raspberrypi systemd[1]: Starting LSB: pihole-FTL daemon...
Jul 19 09:44:21 raspberrypi pihole-FTL[28572]: Not running
Jul 19 09:44:21 raspberrypi su[28594]: (to pihole) root on none
Jul 19 09:44:21 raspberrypi su[28594]: pam_unix(su:session): session opened for user pihole by (uid=0)
Jul 19 09:44:22 raspberrypi pihole-FTL[28572]: FTL started!
Jul 19 09:44:22 raspberrypi su[28594]: pam_unix(su:session): session closed for user pihole
Jul 19 09:44:22 raspberrypi systemd[1]: Started LSB: pihole-FTL daemon.

What entries are in /var/log/syslog related to the cron gravity update this morning?

none, I've disabled cron messages in syslog.

*.*;cron,auth,authpriv.none		-/var/log/syslog

All I have is the mail, send by the system (cron job with cat /var/log/pihole_updateGravity.log)

As I initially mentioned, I duplicated the cron job (changed the time) and had cron execute it again, same error message. I did (just now) this again, adding the line

echo "$PATH"

output (in the log file):

/usr/bin:/bin:/usr/local/bin/

which confirms my assumption the script (when executed with cron) doesn't know where to find service (/usr/sbin)

when running pihole -g from the command line, it does know this:

echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games

There is something different in your install than in mine. I don't see this error in any of my running Pi-hole installs, which leads me to believe it isn't a generic script problem but a local problem.

OS version Raspberry Pi OS (32-bit) Lite, Minimal image based on Debian Buster, installed 3 days ago.

uname -a
Linux raspberrypi 4.19.118-v7+ #1311 SMP Mon Apr 27 14:21:24 BST 2020 armv7l GNU/Linux

I never change the $PATH variable, so the setting is out of the box, I make it a habit to always use the full path to binaries, when writing scripts OR cd into the directory before executing.

I can easely solve this problem, by changing the script (sed) OR create a symbolic link in /usr/local/bin/ for service.

Solves my problem, doesn't solve it for other users, who might not even be aware there is a problem.

The output of the $PATH variable, when executed with cron says it all, I don't know what has changed, I'm just sure I didn't do it.

I cannot duplicate this problem on that same OS.

I'm experiencing exactly the same. Installed pi 5.0 (new installation, latest branch, x86 setup on a raspberry pi 4b 8gb) about 2 weeks ago and updated last week to 5.1.1
Had some issues with stats but a cleanup from the browser sorted that out.

  1. Now you have a confirmed second case, thanks for that @beerns

  2. A user on another dutch forum makes the following remark, translation:

For some reason my pihole triggers the service pihole-FTL restart section of the script, however, there are 3 choices there:

  • svc="kill -SIGRTMIN $(pidof ${resolver})"
  • svc="killall -s SIGHUP ${resolver}"
  • svc="service ${resolver} restart"

Unfortunately, I haven't been able to figure out why my pihole makes the service choice, while your pihole obviously makes another choice.

Given the fact that cron executes the script with the path /usr/bin:/bin:/usr/local/bin/ (see earlier), the script will only fail in one of three cases:

  • kill -> which kill -> /bin/kill -> success (on the path used by cron)
  • killall -> which killall -> /usr/bin/killall -> success (on the path used by cron)
  • service -> which service -> /usr/sbin/service -> fail (NOT on the path used by cron)

All I do, before cron executes the weekly gravity update (I do this on Saturday 23h00, success or failure is registered) is download some lists locally and process them, to make them pihole compatible, examples shallalist, using md5 checksum to ensure integrity and duckduckgo trackers.

Looking into this for more than a day now, I'm convinced this is a real problem, that some users may or may not notice.

For now, I solved the problem, by creating a symbolic link for service:

sudo ln -s /usr/sbin/service /usr/local/bin/service

Another possible (better?) solution would be to change the code:

…
else
  # A full restart has been requested
  serviceCommand=$(which service)
  svc="${serviceCommand} ${resolver} restart"
  str="Restarting DNS server"
fi
…

It would probably be wise to do the same for the kill and killall binaries, this to avoid future problems.

@DL6ER for the crash. They have a few bug fixes and are about to release v5.1.1 at any point. Maybe your bug is already covered

Not the final answer but a step higher: Gravity does a full restart if it doesn't find a pidof pihole.

If the process is running, it just reload

if I just run pidof pihole-FTL, I get a valid process identifier, I have no idea why it would not find this when cron runs the script (which pidof -> /bin/pidof -> on the path used by cron)

Do you at least agree there might be a problem on some systems, reason currently unknown?

Of course - otherwise I wouldn't have invested time to track this down :slight_smile:

I'm just not the one to decide if this is a user-generated issue or a bug. But you provided already a good hint/workaround/solution the devs can look into.

33 3 * * 7 root PATH="$PATH:/usr/local/bin/:/usr/sbin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log