Pihole breaks repositories after reboot

I'm running Raspbian Jesse 2017-01-11

It works normally, I can install everything I want with no issues. (Everything being installing apps with apt-get)

After I install Pihole, Everything still works, I can still download and install apps no problems. After I reboot, I can't install ANYTHING.. Nothing. I can't even update the raspberry with "sudo apt-get upgrade". I get stuck with the message "Cannot resolve mirror.web-ster.com"

This ONLY.... ONLY happens after I install pihole. And after the reboot of that install. It works fine UNTIL A REBOOT.

this hasn't happened prior to the new updates of Pihole. I have it running on my personal pihole for my home. Updating that seems to be fine and everything works well.. My problem lies on a fresh install. And I only get resolve failures after I restart after installing pihole on a fresh version of Raspbian...

So long story short. Everything works just fine until I install the new version of pihole on a freshly flashed image of Raspbian. Then I can't connect and install anything at all. I can successfully reproduce this problem every single time on 3 different Pi's I have here.

And in case anyone is wondering, someone is having me do this for them after seeing mine in action.. Cause this thing rocks and everyone needs one.

Thanks in advance. I can answer any extra questions about my issue.

Can you run pihole -d and follow the prompts when you have a locked condition?

You probably will not be able to upload the debug log, but you will get a local path to the log file and we need to see that to start the debug process.

This log if from the other day. I can generate one from today if you'd like, just let me know.

Also, the 65.4.154.xx base IP is my custom IP for my network. That's manually configured by me. Changed from the normal 192.168.1.xx

Detecting Installed Package Versions:

Pi-hole: v2.11.2
WebUI: v2.3.1
lighttpd/1.4.35
PHP 5.6.29-0+deb8u1 (cli) (built: Dec 17 2016 06:04:43) 

Detecting existence of /etc/pihole/setupVars.conf:

PIHOLE_INTERFACE=eth0
IPV4_ADDRESS=65.4.154.51
IPV6_ADDRESS=
PIHOLE_DNS_1=8.8.8.8
PIHOLE_DNS_2=8.8.4.4
QUERY_LOGGING=true
WEBPASSWORD=


Detecting installed OS Distribution

PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Checking processor variety

armv6l

IP Address Information

::1/128
fe80::272b:70b3:a8f5:94f8/64
127.0.0.1/8
65.4.154.51/8
--- 65.4.154.41 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.441/0.484/0.545/0.050 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 14.113/16.467/18.811/1.917 ms

Daemon Process Information

Found user root with process apache2
Failure: IPv4 Port not in use

Daemon Process Information

Found user dnsmasq with process dnsmasq
Found user dnsmasq with process dnsmasq

Processes Check


lighttpd
 processes status:
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled)
   Active: failed (Result: exit-code) since Tue 2017-01-17 08:14:04 UTC; 11h ago
  Process: 674 ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (code=exited, status=255)
  Process: 616 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 674 (code=exited, status=255)

Jan 17 08:14:01 raspberrypi lighttpd[616]: Syntax OK
Jan 17 08:14:01 raspberrypi systemd[1]: Started Lighttpd Daemon.
Jan 17 08:14:04 raspberrypi lighttpd[674]: 2017-01-17 08:14:01: (network.c.409) can't bind to port:  80 Address already in use
Jan 17 08:14:04 raspberrypi systemd[1]: lighttpd.service: main process exited, code=exited, status=255/n/a
Jan 17 08:14:04 raspberrypi systemd[1]: Unit lighttpd.service entered failed state.

dnsmasq
 processes status:
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
   Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled)
  Drop-In: /run/systemd/generator/dnsmasq.service.d
           └─50-dnsmasq-$named.conf, 50-insserv.conf-$named.conf
   Active: active (running) since Tue 2017-01-17 08:13:53 UTC; 11h ago
 Main PID: 413 (dnsmasq)
   CGroup: /system.slice/dnsmasq.service
           └─413 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service

Jan 17 08:13:50 raspberrypi systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Jan 17 08:13:50 raspberrypi dnsmasq[363]: dnsmasq: syntax check OK.
Jan 17 08:13:53 raspberrypi systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.


Resolver Functions Check

Resolution of doubleclick.com from Pi-hole:

; <<>> DiG 9.9.5-9+deb8u9-Raspbian <<>> doubleclick.com @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37944
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;doubleclick.com.		IN	A

;; ANSWER SECTION:
doubleclick.com.	300	IN	A	65.4.154.51

;; Query time: 5 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jan 17 19:59:08 UTC 2017
;; MSG SIZE  rcvd: 60

Resolution of doubleclick.com from 8.8.8.8:

; <<>> DiG 9.9.5-9+deb8u9-Raspbian <<>> doubleclick.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62688
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;doubleclick.com.		IN	A

;; ANSWER SECTION:
doubleclick.com.	299	IN	A	172.217.3.206

;; Query time: 23 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Jan 17 19:59:08 UTC 2017
;; MSG SIZE  rcvd: 60

Pi-hole dnsmasq specific records lookups
Cache Size:
"10000"
Upstream Servers:
"8.8.4.4#53 14 0" "8.8.8.8#53 16 2"


Detecting existence of /etc/lighttpd/lighttpd.conf:

server.modules = (
"mod_access",
"mod_accesslog",
"mod_auth",
"mod_expire",
"mod_compress",
"mod_redirect",
"mod_setenv",
"mod_rewrite"
)
server.document-root        = "/var/www/html"
server.error-handler-404	= "pihole/index.php"
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
server.port                 = 80
accesslog.filename			= "/var/log/lighttpd/access.log"
accesslog.format			= "%{%s}t|%V|%r|%s|%b"
index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
compress.cache-dir          = "/var/cache/lighttpd/compress/"
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
$HTTP["url"] =~ "^/admin/" {
setenv.add-response-header = (
"X-Pi-hole" => "The Pi-hole Web interface is working!",
"X-Frame-Options" => "DENY"
)
}
url.rewrite = ( "^(?!/admin/).*\.js$"  => "pihole/index.js"   )
$HTTP["url"] =~ "^(?!/admin)/.*" {
setenv.add-response-header = ( "X-Pi-hole" => "A black hole for Internet advertisements." )
}
include_shell "cat external.conf 2>/dev/null"


Detecting existence of /var/log/lighttpd/error.log:

2017-01-17 07:29:28: (log.c.164) server started
2017-01-17 07:31:24: (server.c.1558) server stopped by UID = 0 PID = 1
2017-01-17 07:31:27: (log.c.164) server started
2017-01-17 08:13:36: (server.c.1558) server stopped by UID = 0 PID = 1


Detecting existence of /etc/dnsmasq.conf:

conf-dir=/etc/dnsmasq.d


Detecting existence of /etc/dnsmasq.d/01-pihole.conf:

addn-hosts=/etc/pihole/gravity.list
addn-hosts=/etc/pihole/local.list
domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
interface=eth0
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=300
log-async


Detecting existence of /etc/pihole/whitelist.txt:

raw.githubusercontent.com
mirror1.malwaredomains.com
sysctl.org
zeustracker.abuse.ch
s3.amazonaws.com
hosts-file.net


Detecting existence of /etc/pihole/blacklist.txt:

/etc/pihole/blacklist.txt not found!

Detecting existence of /etc/pihole/adlists.list:

/etc/pihole/adlists.list not found!

Analyzing gravity.list

/etc/pihole/gravity.list is 95120 /etc/pihole/gravity.list lines long.

pihole.log

Okay, that shows that the Pi-hole is able to contact the internet via IP address, so we know that the Pi has at least basic connectivity.

Apache is fighting with lighttpd, but that should just affect the web interface and not the resolver.

Here we can see that the dnsmasq process is able to turn IP addresses into names locally and it is serving the Pi-hole address for the ad domain.

And this shows that the Pi is able to query Google for the real IP address.

Lastly those records are from the dnsmasq process and show that it is loading the proper configuration files, and it's using Google for the upstream resolution.

So other than the Apache2 issue, things look good from your debug log. If the Pi isn't resolving locally for other processes, there may be a misconfiguration in the /etc/resolv.conf file, can we take a look at that next?

Sure thing, I reinstalled earlier and it doesn't have apache on it, it has nothing on it at all except for pihole. And I get the same issue, I can on both access the webUI even though it said failed cause of apache. Kinda weird..

If you want that debug from today with a clean install, just let me know.

Here is the resolv.conf

Generated by resolvconf
nameserver 127.0.0.1

looks the same here as it does on my working pi?

This is the message I get when I try to install anything or run sudo apt-get upgrade

pi@raspberrypi:~ $ sudo apt-get install htop
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ltrace
The following NEW packages will be installed:
  htop
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 68.8 kB of archives.
After this operation, 135 kB of additional disk space will be used.
Err http://mirrordirector.raspbian.org/raspbian/ jessie/main htop armhf 1.0.3-1
  Cannot initiate the connection to mirror.web-ster.com:80 (2604:d200::39). - connect (101: Network is unreachable) [IP: 2604:d200::39 80]
E: Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/h/htop/htop_1.0.3-1_armhf.deb  
Cannot initiate the connection to mirror.web-ster.com:80 (2604:d200::39). - connect (101: Network is unreachable) [IP: 2604:d200::39 80] 
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mis                                                                                                                               sing?

I am able to install all of these before a restart after installing pihole. Only before a restart though. After, I can't install anything.

Those are IPv6 adresses that it's trying to reach, but from your debug log, there are no IPv6 IP's assigned to the Pi-hole? What does ip a or ip link look like?

ip a or ip link? My personal pi has an ipv6 address assigned to it. But that came only after the last updates.. I didn't configure that.

I have the pi itself configured for ipv4.

From the device that is having the problem with apt-get can you run the following command:

ip route get 8.8.8.8
ip route get 2001:4860:4860::8888
ip addr

I think this may be an IPv6 routing issue, which would have been coincidental since IPv6 would be something that is handled by your ISP and not by us. But we'll see if we can find the issue.

pi@raspberrypi:~ $ ip route get 8.8.8.8
8.8.8.8 via 65.4.154.41 dev eth0 src 65.4.154.51
cache
pi@raspberrypi:~ $ ip route get 2001:4860:4860::8888
unreachable 2001:4860:4860::8888 from :: dev lo table unspec proto kernel src fe80::272b:70b3: a8f5:94f8 metric 4294967295 error -101
pi@raspberrypi:~ $ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:70:3e:ee brd ff:ff:ff:ff:ff:ff
inet 65.4.154.51/8 brd 65.255.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::272b:70b3:a8f5:94f8/64 scope link
valid_lft forever preferred_lft forever
pi@raspberrypi:~ $

Thanks again for helping with this. You're responding much faster than I'm expecting you to. Am grateful

Also, IPV6 is disabled on my rotuer. I'm running Shibby Tomato on an Asus AC68U.

Okay, that looks like the client is being instructed to use IPv6 to contact the mirror, however it doesn't have a route to get out. That's the unreachable part of the route response. It's got a local IPv6 address that won't route to the internet, the fe80 address, but I'm not sure why that device is trying to contact the mirrors via IPv6 when it's not ever going to get a response.

I have a vague memory of there being IPv6 issues with Raspbian in the past, but I'm don't remember the full situation.

What does just a plain ip route show for the devices routing table?

Is this what you're asking for??

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 65.4.154.41 0.0.0.0 UG 202 0 0 eth0
65.0.0.0 0.0.0.0 255.0.0.0 U 202 0 0 eth0
pi@raspberrypi:~ $

Or probably this

default via 65.4.154.41 dev eth0 metric 202
65.0.0.0/8 dev eth0 proto kernel scope link src 65.4.154.51 metric 202
pi@raspberrypi:~ $

Yes, I have no idea why your device is trying to get out via IPv6 to the mirror when there is no way for it to do so. Let me do some research and see if I can find anything that would help, or why there would be this situation with the Pi. Rasbian's mirror director has been having issues lately, but nothing that would return an IPv6 address if its not queried for.

Was the 65.4.154.51/8 network allocation provided to you by your ISP? That is a public IP netblock and I can't ever remember someone getting an entire /8 block assigned to them. Did you mean for this to be configured with a public IP address?

The pi I'm currently trying to configure is connected to my network here that already has a working pi acting as the DNS.

I don't know if that helps, but it didn't matter before if there were 2 pi's on the network..

This is just my local ip for the devices on my network. The number is something I personally assigned as my local range.

I don't like using the standard ip for my local network and only use the 10.0.0.1 base of ip's for systems I work on.

That may cause issues, since that address block is allocation to North America, you might lose routing connectivity to a large number of hosts. But if it worked prior to an update, lets see if its related to the IP stack.

The following command should force apt-get to run over IPv4 and that will help check to see if there is a stack issue or another place to cross off the list.

sudo apt-get update -o Acquire::ForceIPv4=true

I ran that command, and tried to apt-get htop. No go.

 Acquire::ForceIPv4=true

That only works for the command issued at the same time. If it successfully updated, then try instead an

sudo apt-get install htop -o Acquire::ForceIPv4=true

And see if that installs it.

pi@raspberrypi:/etc/dnsmasq.d $ sudo apt-get install htop -o Acquire::ForceIPv4=true
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  ltrace
The following NEW packages will be installed:
  htop
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 68.8 kB of archives.
After this operation, 135 kB of additional disk space will be used.
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main htop armhf 1.0.3-1 [68.8 kB]
Fetched 68.8 kB in 0s (92.8 kB/s)
Selecting previously unselected package htop.
(Reading database ... 34104 files and directories currently installed.)
Preparing to unpack .../htop_1.0.3-1_armhf.deb ...
Unpacking htop (1.0.3-1) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up htop (1.0.3-1) ...

pi@raspberrypi:~ $ sudo apt-get install htop -o Acquire::ForceIPv4=true
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
ltrace
The following NEW packages will be installed:
htop
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 68.8 kB of archives.
After this operation, 135 kB of additional disk space will be used.
Err Index of /raspbian jessie/main htop armhf 1.0.3-1
Could not connect to mirror.web-ster.com:80 (65.182.224.39). - connect (113: No route to host)
E: Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/h/htop/htop_1.0.3-1_armhf.deb Could not connect to mirror.web-ster.com:80 (65.182.224.39). - connect (113: No route to host)

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
pi@raspberrypi:~ $