Odd circumstance, DHCP good, DNS not good, admin web interface no good

JoelCarpenter · January 24, 2020, 10:34am

Please follow the below template, it will help us to help you!

Expected Behaviour:

Admin web interface to load and DNS to resolve

Actual Behaviour:

Serves DHCP fine. Doesn't resolve DNS (nslookup). Admin web interface not loading, but can ssh in and get a ping response from other network devices.

I have rebooted several times and restarted the FTL service from SSH.

Debug Token:

https://tricorder.pi-hole.net/5u72x07qo0

Would love it if the debug log gave a vague hint on this one, been troubleshooting for hours. Can't see what's changed.

Thanks in advance

jfb · January 24, 2020, 3:54pm

From your debug log - there is a mismatch between the IP of the Pi-Hole host, and the IP you have in the Pi-Hole configuration. As a result, clients cannot reach the Pi-Hole on the LAN for DNS traffic.

pihole -r and select "reconfigure" will let you put the Pi-Hole on the active IP and interface.

*** [ DIAGNOSING ]: Networking
[✓] IPv4 address(es) bound to the eth0 interface:
   10.0.0.254/8 does not match the IP found in /etc/pihole/setupVars.conf (https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127)

*** [ DIAGNOSING ]: Name resolution (IPv4) using a random blocked domain and a known ad-serving domain
[✓] bowlingle.com is 0.0.0.0 via localhost (127.0.0.1)
[✗] Failed to resolve bowlingle.com via Pi-hole (10.0.0.254)

Pi-Hole checks for running processes, and in this case reports that lighttpd is failed (not running). The debug log also outputs the lighttpd error log, but in your case the log is empty since lighttpd is not running.

*** [ DIAGNOSING ]: Pi-hole processes
[✗] lighttpd daemon is failed

*** [ DIAGNOSING ]: contents of /var/log/lighttpd

-rw-r--r-- 1 root root 0 Jan 23 23:00 /var/log/lighttpd/error.log

Let's take a look at any lighttpd related conflicts or errors. What are the outputs of the following commands from the Pi terminal:

sudo netstat -nltup | grep 'Proto\|:53 \|:5053 \|:5353 \|:8953 \|:67 \|:80 \|:471'

sudo systemctl status --full --no-pager lighttpd

JoelCarpenter · January 24, 2020, 10:46pm

Thanks so much for the reply:

So.....

PiHole -r reconfigure and repair both give the same output (I don't get the option to reconfig the IP address etc):

root@sizzlepi:~# pihole -r
  [✓] Root user check
        .;;,.
        .ccccc:,.
         :cccclll:.      ..,,
          :ccccclll.   ;ooodc
           'ccll:;ll .oooodc
             .;cll.;;looo:.
                 .. ','.
                .',,,,,,'.
              .',,,,,,,,,,.
            .',,,,,,,,,,,,....
          ....''',,,,,,,'.......
        .........  ....  .........
        ..........      ..........
        ..........      ..........
        .........  ....  .........
          ........,,,,,,,'......
            ....',,,,,,,,,,,,.
               .',,,,,,,,,'.
                .',,,,,,'.
                  ..'''.
  [i] Existing PHP installation detected : PHP version 7.0.33-0ubuntu0.16.04.9
  [i] Reconfigure option selected
  [✓] Disk space check
  [✓] Update local cache of available packages
  [i] Checking apt-get for upgraded packages...E: Unmet dependencies. Try using -f.
  [✓] Checking apt-get for upgraded packages... up to date!
  [i] Installer Dependency checks...
  [✓] Checking for apt-utils
  [✓] Checking for dialog
  [✓] Checking for debconf
  [✓] Checking for dhcpcd5
  [✓] Checking for git
  [i] Checking for iproute (will be installed)
  [✓] Checking for whiptail

Although with the reconfigure I do briefly see the configuration screen pop and disappear in an instant.

Maybe it's time to re-flash this puppy! Unfortunately I've done that a few times lately and I'm starting to suspect a hardware error somewhere around the place.

Interestingly, if I edit /etc/pihole/setupVars.conf the listed IP is the same as in the debug file.

Here are the outputs of the two commands you suggested:

root@sizzlepi:~# sudo netstat -nltup | grep 'Proto\|:53 \|:5053 \|:5353 \|:8953 \|:67 \|:80 \|:471'

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:4711          0.0.0.0:*               LISTEN      1086/pihole-FTL
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1197/dnsmasq
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      1086/pihole-FTL
tcp6       0      0 ::1:4711                :::*                    LISTEN      1086/pihole-FTL
tcp6       0      0 ::1:53                  :::*                    LISTEN      1086/pihole-FTL
tcp6       0      0 fe80::e098:f4ff:fedb:53 :::*                    LISTEN      1086/pihole-FTL
udp        0      0 127.0.1.1:53            0.0.0.0:*                           1197/dnsmasq
udp        0      0 127.0.0.1:53            0.0.0.0:*                           1086/pihole-FTL
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1086/pihole-FTL
udp6       0      0 ::1:53                  :::*                                1086/pihole-FTL
udp6       0      0 fe80::e098:f4ff:fedb:53 :::*                                1086/pihole-FTL

sudo systemctl status --full --no-pager lighttpd

● lighttpd.service - Lighttpd Daemon

   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2020-01-25 08:17:16 AEDT; 1h 23min ago
  Process: 1022 ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (code=exited, status=255) 
  Process: 945 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
Main PID: 1022 (code=exited, status=255)
Jan 25 08:17:14 sizzlepi systemd[1]: Starting Lighttpd Daemon...
Jan 25 08:17:15 sizzlepi lighttpd[945]: Syntax OK
Jan 25 08:17:15 sizzlepi systemd[1]: Started Lighttpd Daemon.
Jan 25 08:17:16 sizzlepi lighttpd[1022]: 2020-01-25 08:17:15: (log.c.118) opening errorlog '/var/log/lighttpd/error.log' failed: Permission denied
Jan 25 08:17:16 sizzlepi systemd[1]: lighttpd.service: Main process exited, code=exited, status=255/n/a
Jan 25 08:17:16 sizzlepi systemd[1]: lighttpd.service: Unit entered failed state.
Jan 25 08:17:16 sizzlepi systemd[1]: lighttpd.service: Failed with result 'exit-code'.

Thanks again... point me to the donate button - I think it's time...

deHakkelaar · January 25, 2020, 12:36am

You so much messed up the place ... big

If above one is not installed:

apt policy iproute

I suspect something wrong with your apt sources:

sudo apt update

You've managed to get dnsmasq started alongside pihole-FTL wich will conflict fighting over port 53.
Disable dnsmasq from starting at boot:

sudo systemctl disable dnsmasq

sudo systemctl stop dnsmasq

sudo systemctl restart pihole-FTL

And check again if dnsmasq is not listed now:

sudo netstat -nltup | grep 'Proto\|:53 \|:5053 \|:5353 \|:8953 \|:67 \|:80 \|:471'

Above one says permissions are wrong.
Make sure the file exists:

sudo touch /var/log/lighttpd/error.log

Owner should be www-data for most distro's (or else the owner is most likely lighttpd):

sudo chown www-data:www-data /var/log/lighttpd/error.log

And permissions 644:

sudo chmod 644 /var/log/lighttpd/error.log

Restart:

sudo systemctl restart lighttpd

And check again:

sudo systemctl status --full --no-pager lighttpd

EDIT: typo in sudo systemctl disable dnsmasq

JoelCarpenter · January 25, 2020, 1:01am

Awesome, this is a great lesson, thank you

So, disabling and/or stopping dnsmasq returns an error:

root@sizzlepi:~# sudo systemctl stop dnsmasq

Failed to stop dnsmasq.service: Unit dnsmasq.service not loaded.

root@sizzlepi:~# sudo systemctl disable dnsmasq

Failed to execute operation: No such file or directory

And yet, it still exists in the output from the netstat command under port 53 here:

tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1191/dnsmasq

and here:

udp        0      0 127.0.1.1:53            0.0.0.0:*                           1191/dnsmasq

The lighttpd stuff ultimately resulted in:

root@sizzlepi:~# sudo systemctl status --full --no-pager lighttpd

● lighttpd.service - Lighttpd Daemon

   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)

   Active: failed (Result: exit-code) since Sat 2020-01-25 11:58:25 AEDT; 16s ago

  Process: 3296 ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (code=exited, status=255)

  Process: 3288 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)

Main PID: 3296 (code=exited, status=255)

Jan 25 11:58:24 sizzlepi systemd[1]: Starting Lighttpd Daemon...

Jan 25 11:58:25 sizzlepi lighttpd[3288]: Syntax OK

Jan 25 11:58:25 sizzlepi systemd[1]: Started Lighttpd Daemon.

Jan 25 11:58:25 sizzlepi lighttpd[3296]: 2020-01-25 11:58:25: (log.c.118) opening errorlog '/var/log/lighttpd/error.log' failed: Permission denied

Jan 25 11:58:25 sizzlepi systemd[1]: lighttpd.service: Main process exited, code=exited, status=255/n/a

Jan 25 11:58:25 sizzlepi systemd[1]: lighttpd.service: Unit entered failed state.

Jan 25 11:58:25 sizzlepi systemd[1]: lighttpd.service: Failed with result 'exit-code'.

I'm going to say that's not a good result

So..... I really stuffed up the place didn't I? - not entirely sure how, mind you.

deHakkelaar · January 25, 2020, 1:10am

Am not sure what distro you have running and if it runs with a desktop.
If it runs with a desktop (Gnome, KDE etc), most likely dnsmasq gets started as part of some network manager.
I wouldn't install Pi-hole with a desktop as it eats resources (RAM,CPU etc) , and you get into troubles like this whit a bunch of software installed that ads to the security risk factor and complexity.

To proceed, you would really need to get your apt sources on track which you did not post:

Ow ps. could you enclose your code output with this
button while posting here pls?
Makes reading alot easier.

JoelCarpenter · January 25, 2020, 1:18am

Running Ubuntu Bionic 18.04.3 LTS

Been running upgrades and updates - iproute says not installed and apt-get said it's not available so it offered me iproute2, which I installed.

All of a sudden nslookups are working via the pihole... I'm SO confused

Just got to get lighttpd back up and running now..

getting there, thanks again

deHakkelaar · January 25, 2020, 1:19am

Better chance now that pihole -r will run.

JoelCarpenter · January 25, 2020, 1:21am

and apologies, the linux is not strong in this one....for the Sudo apt update I get this:

root@sizzlepi:~# sudo apt update
Hit:1 http://ports.ubuntu.com bionic InRelease
Hit:2 http://ports.ubuntu.com bionic-security InRelease
Hit:3 http://ports.ubuntu.com bionic-updates InRelease
Hit:4 http://ports.ubuntu.com bionic-backports InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
456 packages can be upgraded. Run 'apt list --upgradable' to see them.

but I can't work out how to run the updates.. I've tried 'apt-get uprade' to no avail.....

deHakkelaar · January 25, 2020, 1:22am

Looks good.
iproute was the issue for pihole -r

Would have to check the output what goes wrong.

Ps. thats a typo right ?

apt-get upgrade

JoelCarpenter · January 25, 2020, 1:27am

Yep, typo!

PiHole -r and then reconfigure worked... happy days!

Managed to get my apt-upgrades running somehow, it's currently working through several hundred packages... I'll report back.

deHakkelaar · January 25, 2020, 1:28am

Try purge lighttpd and reset/default everyting:

JoelCarpenter · January 25, 2020, 1:56am

Alrighty, we're getting somewhere. Lighttpd is back up and running.

This has been great, thank you so much.

The last remaining thing (and this is a persistent gap in my knowledge) is lining up the IP address of the PiHole in all relevant places. The DHCP server on the Pi seems to now be giving out 10.0.0.64 as itself but the DNS address is 10.0.0.254 (my intended address) which is reflected on the web interface and ipconfig on my PC but the DHCP address shows as .64 from that ipconfig result.

When I do an 'ip address show' I get on eth0:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether e2:98:f4:db:63:94 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.64/24 brd 10.0.0.255 scope global eth0
    inet 10.0.0.254/8 brd 10.255.255.255 scope global eth0
    inet6 fe80::e098:f4ff:fedb:6394/64 scope link
       valid_lft forever preferred_lft forever

Why the two addresses, do I need to manually edit the DHCP address, surely it can also exist on the .254 address? Or not?

I also have another client (iPhone) that receives .64 as DNS and a whacky net mask of 255.0.0.0

deHakkelaar · January 25, 2020, 2:02am

Do you run a desktop ?
Pi-hole comes with its own network manager called dhcpcd5:

pi@noads:~ $ apt policy dhcpcd5
dhcpcd5:
  Installed: 1:6.11.5-1+rpt7

pi@noads:~ $ tail /etc/dhcpcd.conf
[..]
interface eth0
  static ip_address=10.0.0.2/24
  static routers=10.0.0.1
  static domain_name_servers=127.0.0.1

It could be that your desktop (or even you distro) comes with its own network manager/mangler that also tries to configure networking.

JoelCarpenter · January 25, 2020, 2:03am

Nothing a good reboot can't fix. I seem to have everything back on .254 after a reboot.... I 'think' most of my issues are resolved.

I really appreciate your quick and detailed responses @deHakkelaar - I've learnt heaps and appear to have my network back in shape.

Many thanks

system · February 15, 2020, 2:03am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.