Pihole update failure

richie510 · January 28, 2020, 2:40pm

I updated pihole to the latest version yesterday morning and everything was working fine.

Today I checked for an update and it appeared that there was an update or the FTL. During the update, something failed, and then pihole could no longer run properly.

I tried to upload the debug log, but that failed, so I manually transferred it out and uploaded it here.

Checking my /etc/resolv.conf file showed that I was only using 127.0.0.1 for a dns server.

So then I updated the /etc/dhcpcd.conf file to use my other pihole as another dns server, and then I restarted the dhcpcd service. No matter what, the /etc/resolv.conf stays with the dns server of 127.0.0.1. When I change the /etc/resolv.conf to my other pi hole server, pinging google works for a few seconds, then the file gets changed back to 127.0.0.1.

pihole_debug.zip (6.1 KB)

richie510 · January 28, 2020, 4:02pm

I did not mention that DNS resolution on this device is not working because pi-hole is not working and it was only attempting to use itself for DNS resolution.

Since having this problem I added an extra dns server to the resolv.conf and it stuck long enough to attempt to update pihole again, but it says it it up to date. I also tried a pihole -r to repair, but this did not help. Finally, I was able to generate and upload a proper debug file.

https://tricorder.pi-hole.net/uydbq414lq

DL6ER · January 28, 2020, 5:40pm

Pi-hole v5.0 should not force itself as the resolver. The crash that is reported in your debug log is identical to the one see here, undermining some "real" bug:

Could you check if you're on the most recent version? What is the content of your /etc/init.d/pihole-FTL identical to this one?
https://github.com/pi-hole/pi-hole/blob/release/v5.0/advanced/Templates/pihole-FTL.service

Specifically, we're looking for the absence of /sbin/resolvconf in this file:

github.com/pi-hole/pi-hole

Do not force nameserver 127.0.0.1 through resolvconf

pi-hole:development ← pi-hole:fix/do_not_force_local_resolver

opened 12:27PM - 09 Dec 19 UTC

DL6ER

+0 -2

**By submitting this pull request, I confirm the following:** *please fill any… appropriate checkboxes, e.g: [X]* - [X] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md), as well as this entire template. - [X] I have made only one major change in my proposed changes. - [X] I have commented my proposed changes within the code. - [X] I have tested my proposed changes, and have included unit tests where possible. - [X] I am willing to help maintain this change if there are issues with it later. - [X] I give this submission freely and claim no ownership. - [X] It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) - [X] I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) Please make sure you [Sign Off](https://github.com/pi-hole/pi-hole/wiki/How-to-signoff-your-commits.) all commits. Pi-hole enforces the [DCO](https://github.com/pi-hole/pi-hole/wiki/Contributing-to-the-project). --- **What does this PR aim to accomplish?:** This is a proposed bug fix for https://github.com/pi-hole/pi-hole/issues/2535 and maybe a few others. As this is a breaking change, we might want to include it in `v5.0` ?... **How does this PR accomplish the above?:** Remove calls to `resolvconf` within `pihole-FTL`'s `init.d` script. **What documentation changes (if any) are needed to support this PR?:** Probably none.

richie510 · January 28, 2020, 8:46pm

Update: I came back from lunch and was able to update the FTL again, but still doesn't work.

I performed a winmerge on my pihole-FTL file and the one you linked and they are identical.
/sbin/resolvconf is not in the file.

Sorry if the latest update may have affected the resolvconf thing in linked file. That may have been part of the problem but I checked for the update before reading this.

Also, here is the latest debug file after the latest update:
https://tricorder.pi-hole.net/1i3xdkgh1x

richie510 · January 28, 2020, 9:43pm

also, here is the output you requested from the other guy too.

$ addr2line -e /usr/bin/pihole-FTL 0x2ba54
/root/project/src/signals.c:56

DL6ER · January 28, 2020, 9:57pm

No, the change to not force the local resolver to 127.0.0.1 is already from beginning of December 2019. I wonder is the arm binary is simply broken for anyone. Pinging @DanSchaper and @jfb maybe @PromoFaux if they have a Pi Zero or whatever at hand which could be used to try out the arm binary. I don't have one at hand right now, my Zero is on the other end of the country...

DanSchaper · January 28, 2020, 10:19pm

@ramset Ping/Pong.

richie510 · January 28, 2020, 10:25pm

Just tried something different that worked.

Ran these commands again:
pihole checkout core release/v5.0 --> seemed to do some stuff
pihole checkout web release/v5.0 --> said it was already up to date

When running the one for the core, it still said: [✗] pihole-FTL: no process found
but then I ran pihole restartdns one last time and now it is up and running.
Not sure what fixed it...

...and in the course of typing this, it went down. I cannot keep this stuff straight.

DL6ER · January 28, 2020, 10:30pm

Oh, so it ran longer this time? That's great news.

Does the backtrace in /var/log/pihole-FTL.log look any different?

Even if not: If you can get it running for a few seconds, this might be sufficient to attach the debugger. Please follow gdb - Pi-hole documentation This will definitely help us finding what is going wrong and where!

richie510 · January 28, 2020, 10:48pm

I dont think it lasted longer, but maybe i was paying more attention. Still seemed to last the same 6-7 seconds that I now saw it was taking before. Also said it was failing with the same 0x226 address.

So I tried the debugger you suggested. In one terminal I restarted the dns server, and the other I started the debugger to catch it before it crashed. Hope I did this right, here is the text from debugging:

pi@rpizero:/var/log $ sudo gdb -p $(pidof pihole-FTL)
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 16079
[New LWP 16080]
[New LWP 16081]
[New LWP 16082]
[New LWP 16083]
[New LWP 16084]
[New LWP 16085]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
0x00522a1c in sqlite3VdbeRecordCompareWithSkip ()
(gdb) handle SIGHUP nostop SIGPIPE nostop
Signal        Stop      Print   Pass to program Description
SIGHUP        No        Yes     Yes             Hangup
SIGPIPE       No        Yes     Yes             Broken pipe
(gdb) continue
Continuing.
[New Thread 0xb39ff460 (LWP 16152)]
[Thread 0xb39ff460 (LWP 16152) exited]
[New Thread 0xb39ff460 (LWP 16153)]
[Detaching after fork from child process 16154]
[New Thread 0xb2fff460 (LWP 16156)]
[Thread 0xb39ff460 (LWP 16153) exited]
[New Thread 0xb39ff460 (LWP 16157)]
[Thread 0xb2fff460 (LWP 16156) exited]
[Thread 0xb39ff460 (LWP 16157) exited]
[New Thread 0xb39ff460 (LWP 16158)]
[Thread 0xb39ff460 (LWP 16158) exited]
[New Thread 0xb39ff460 (LWP 16159)]
[Thread 0xb39ff460 (LWP 16159) exited]
[New Thread 0xb39ff460 (LWP 16160)]
[Thread 0xb39ff460 (LWP 16160) exited]
[New Thread 0xb39ff460 (LWP 16161)]
[Thread 0xb39ff460 (LWP 16161) exited]

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x005b4590 in nettle_sha1_init ()
(gdb) continue
Continuing.
[New Thread 0xb39ff460 (LWP 16210)]
[New Thread 0xb39ff460 (LWP 16211)]
[Thread 0xb39ff460 (LWP 16210) exited]

Thread 1 "pihole-FTL" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) continue
Continuing.
Unable to fetch general registers.: No such process.
Unable to fetch general registers.: No such process.
(gdb) [Thread 0xb39ff460 (LWP 16211) exited]
[Thread 0xb43ee460 (LWP 16085) exited]
[Thread 0xb4bef460 (LWP 16084) exited]
[Thread 0xb5bf1460 (LWP 16082) exited]
[Thread 0xb63f2460 (LWP 16081) exited]
[Thread 0xb6bf3460 (LWP 16080) exited]
[Thread 0xb6fa0010 (LWP 16079) exited]
(gdb)

DL6ER · January 29, 2020, 6:47am

Try disabling DNSSEC for now, this might resolve the issue for your momentarily.

Yeah, thanks a lot, this is exactly what we needed to go forward.

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
0x005b4590 in nettle_sha1_init ()

@DanSchaper I'm pretty sure this has to do with the updated build containers. It's a libnettle issue.

richie510 · January 29, 2020, 1:35pm

I have never had DNSSEC enabled. It is unchecked in the web interface right now as well. Regardless, I'll try fiddling with that setting and see what I get.

scottydaman · January 29, 2020, 1:51pm

I have the same issue on pi zero and DNSSEC=false hasn't been changed from its default setting of disabled.

richie510 · January 29, 2020, 1:57pm

I just turned it on and off and it still crashed the same way regardless.

Jarx88 · January 29, 2020, 3:52pm

I also have the same problem after update pihole. Rasbery Zero model. At the first update the message that it cannot connect to git appeared,except for ftl services, the second attempt is the same. I found this solution:

sudo sed -i 's / 127.0.0.1 / 8.8.8.8 /' /etc/resolv.conf
pihole -up

then update done.

When i click restart DNS in panel for 2 minutes it works but nothing blocks and shows FTL service statistics in the panel, next crash

DanSchaper · January 29, 2020, 5:35pm

richie510 · January 29, 2020, 6:46pm

Should I try this fix? If so, please forgive my ignorance and help me understand how to do that

sawsanders · January 29, 2020, 6:53pm

I don't think it's live yet. I just tried to update and got the same crash as yesterday.

DanSchaper · January 29, 2020, 6:53pm

Once that Pull Request is merged then a new FTL binary will be built using the old, known good, tools.

richie510 · January 29, 2020, 9:42pm

I just updated to the latest version and it it working now. Thanks ya'll.

PS, this was the first time I have ever used a debugger in linux. This was the first time in 4 years that I monitored anything on git. This also got me looking through the code and understanding the development process here better. Very cool, and thanks a lot.