Web interface 403 - Forbidden, blocking is working

Expected Behaviour:

Going to the admin page/web interface should show stats etc.

Actual Behaviour:

Going to the admin page shows only a '403 - Forbidden' Message. DNS blocking is working though

Debug Token:

https://tricorder.pi-hole.net/a9g99a9fgq

Something is keeping lighttpd (the web server for the admin page) from running. What are the outputs of the following commands from the Pi terminal:

sudo netstat -nltup | grep 'Proto\|:53 \|:67 \|:80 \|:471'

sudo service lighttpd status

journalctl -u lighttpd

_sudo netstat -nltup | grep 'Proto\|:53 \|:67 \|:80 \|:471'_
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:4711          0.0.0.0:*               LISTEN      590/pihole-FTL
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      534/lighttpd
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      590/pihole-FTL
tcp6       0      0 ::1:4711                :::*                    LISTEN      590/pihole-FTL
tcp6       0      0 :::80                   :::*                    LISTEN      534/lighttpd
tcp6       0      0 :::53                   :::*                    LISTEN      590/pihole-FTL
udp        0      0 0.0.0.0:53              0.0.0.0:*                           590/pihole-FTL
udp6       0      0 :::53                   :::*                                590/pihole-FTL
_sudo service lighttpd status_
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset:
   Active: active (running) since Mon 2019-05-27 14:31:05 BST; 5 days ago
  Process: 513 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.con
 Main PID: 534 (lighttpd)
   CGroup: /system.slice/lighttpd.service
           └─534 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf

May 27 14:31:04 raspberrypi systemd[1]: Starting Lighttpd Daemon...
May 27 14:31:05 raspberrypi systemd[1]: Started Lighttpd Daemon.
_journalctl -u lighttpd_
-- Logs begin at Mon 2019-05-27 14:30:56 BST, end at Sat 2019-06-01 15:07:40 BST
May 27 14:31:04 raspberrypi systemd[1]: Starting Lighttpd Daemon...
May 27 14:31:05 raspberrypi systemd[1]: Started Lighttpd Daemon.

What is the output of these commands?

sudo service lighttpd restart
lighttpd -f /etc/lighttpd/lighttpd.conf -t
ls -al /var/log/lighttpd
cat /var/log/lighttpd/error.log
journalctl -u lighttpd
lighttpd -f /etc/lighttpd/lighttpd.conf -t
Syntax OK
ls -al /var/log/lighttpd
ls: cannot open directory '/var/log/lighttpd': Permission denied
sudo ls -al /var/log/lighttpd
total 324
drwxr-x--- 2 www-data www-data   4096 Jun  2 06:25 .
drwxr-xr-x 7 root     root       4096 Jun  2 06:25 ..
-rw-r--r-- 1 www-data www-data      0 Jun  2 06:25 access.log
-rw-r--r-- 1 www-data www-data   1588 Jun  1 14:09 access.log.1
-rw-r--r-- 1 www-data www-data   9583 Mar 31 21:23 access.log.10.gz
-rw-r--r-- 1 www-data www-data  12262 Mar 24 06:25 access.log.11.gz
-rw-r--r-- 1 www-data www-data   1070 Mar 19 18:01 access.log.12.gz
-rw-r--r-- 1 www-data www-data  10885 May 27 06:25 access.log.2.gz
-rw-r--r-- 1 www-data www-data   2126 May 15 15:34 access.log.3.gz
-rw-r--r-- 1 www-data www-data   1249 May  9 14:31 access.log.4.gz
-rw-r--r-- 1 www-data www-data 121001 May  4 19:12 access.log.5.gz
-rw-r--r-- 1 www-data www-data  24163 Apr 26 12:26 access.log.6.gz
-rw-r--r-- 1 www-data www-data   2491 Apr 21 06:25 access.log.7.gz
-rw-r--r-- 1 www-data www-data   9227 Apr 15 19:07 access.log.8.gz
-rw-r--r-- 1 www-data www-data  49399 Apr  7 06:25 access.log.9.gz
-rw-r--r-- 1 www-data www-data    194 Jun  2 10:43 error.log
-rw-r--r-- 1 www-data www-data    436 May 27 14:31 error.log.1
-rw-r--r-- 1 www-data www-data     90 Mar 25 06:25 error.log.10.gz
-rw-r--r-- 1 www-data www-data    232 Mar 24 06:25 error.log.11.gz
-rw-r--r-- 1 www-data www-data     89 Mar 11 06:25 error.log.12.gz
-rw-r--r-- 1 www-data www-data    486 May 23 17:35 error.log.2.gz
-rw-r--r-- 1 www-data www-data     90 May 13 06:25 error.log.3.gz
-rw-r--r-- 1 www-data www-data     89 May  5 06:25 error.log.4.gz
-rw-r--r-- 1 www-data www-data     90 Apr 29 06:25 error.log.5.gz
-rw-r--r-- 1 www-data www-data     90 Apr 21 06:25 error.log.6.gz
-rw-r--r-- 1 www-data www-data    320 Apr 16 19:00 error.log.7.gz
-rw-r--r-- 1 www-data www-data     89 Apr  7 06:25 error.log.8.gz
-rw-r--r-- 1 www-data www-data    244 Apr  4 15:29 error.log.9.gz

cat /var/log/lighttpd/error.log
cat: /var/log/lighttpd/error.log: Permission denied
sudo cat /var/log/lighttpd/error.log
2019-06-02 06:25:02: (server.c.1534) logfiles cycled UID = 0 PID = 9511
2019-06-02 10:43:39: (server.c.1828) server stopped by UID = 0 PID = 1
2019-06-02 10:43:40: (log.c.217) server started

journalctl -u lighttpd
-- Logs begin at Mon 2019-05-27 14:30:56 BST, end at Sun 2019-06-02 10:43:40 BST
May 27 14:31:04 raspberrypi systemd[1]: Starting Lighttpd Daemon...
May 27 14:31:05 raspberrypi systemd[1]: Started Lighttpd Daemon.
Jun 02 10:43:39 raspberrypi systemd[1]: Stopping Lighttpd Daemon...
Jun 02 10:43:39 raspberrypi systemd[1]: Stopped Lighttpd Daemon.
Jun 02 10:43:39 raspberrypi systemd[1]: Starting Lighttpd Daemon...
Jun 02 10:43:40 raspberrypi systemd[1]: Started Lighttpd Daemon.

Check the error log while accessing the admin page:

sudo tailf /var/log/lighttpd/error.log

Opening the admin page does not lead to additions to the error file, as shown below. This is after muttiple refreshes of the page, also including ctrl+F5 and also in multiple browsers (chrome and edge).

sudo tailf /var/log/lighttpd/error.log
2019-06-02 06:25:02: (server.c.1534) logfiles cycled UID = 0 PID = 9511
2019-06-02 10:43:39: (server.c.1828) server stopped by UID = 0 PID = 1
2019-06-02 10:43:40: (log.c.217) server started

Anything showing tailing the access.log while browsing to the admin page ?
Below how mine looks like:

pi@noads:~ $ sudo tailf /var/log/lighttpd/access.log
1559601604|pi.hole|GET /admin/ HTTP/1.1|200|26200
1559601604|pi.hole|GET /admin/img/logo.svg HTTP/1.1|200|1649
1559601604|pi.hole|GET /admin/style/vendor/font-awesome-5.6.3/webfonts/fa-solid-900.woff2 HTTP/1.1|200|79100
1559601605|pi.hole|GET /admin/style/vendor/font-awesome-5.6.3/webfonts/fa-brands-400.woff2 HTTP/1.1|200|74288
1559601605|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601605|pi.hole|GET /admin/api.php?overTimeData10mins HTTP/1.1|200|4477
1559601605|pi.hole|GET /admin/api.php?overTimeDataClients&getClientNames HTTP/1.1|200|5211
1559601605|pi.hole|GET /admin/api.php?summaryRaw&getQuerySources&topClientsBlocked HTTP/1.1|200|855
1559601605|pi.hole|GET /admin/api.php?summaryRaw&topItems HTTP/1.1|200|1027
1559601605|pi.hole|GET /admin/api.php?getQueryTypes HTTP/1.1|200|102
1559601605|pi.hole|GET /admin/api.php?getForwardDestinations HTTP/1.1|200|138
1559601606|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601607|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601608|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601609|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601610|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601611|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601612|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601613|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601614|pi.hole|GET /admin/api.php?summary HTTP/1.1|200|515
1559601615|pi.hole|GET /admin/api.php?summaryRaw&getQuerySources&topClientsBlocked HTTP/1.1|200|855
[..]

What is the output of these commands?

stat /var
stat /var/www
stat /var/www/html
stat /var/www/html/admin
ls -al /var/www/html/admin

Replying to Mcat12 first:

stat /var
  File: /var
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d    Inode: 71281       Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-11-13 14:02:00.773289306 +0000
Modify: 2018-12-15 16:00:25.722945645 +0000
Change: 2018-12-15 16:00:25.722945645 +0000
 Birth: -
stat /var/www
  File: /var/www
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d    Inode: 127845      Links: 4
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-15 16:00:25.722945645 +0000
Modify: 2018-12-15 16:01:11.322764676 +0000
Change: 2018-12-15 16:01:11.322764676 +0000
 Birth: -
stat /var/www/html
  File: /var/www/html
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d    Inode: 127846      Links: 4
Access: (0775/drwxrwxr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2018-12-15 16:00:25.722945645 +0000
Modify: 2019-05-21 19:13:55.804545395 +0100
Change: 2019-05-27 14:19:18.145646256 +0100
 Birth: -
stat /var/www/html/admin
  File: /var/www/html/admin
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d    Inode: 127727      Links: 7
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-05-21 19:13:39.864566082 +0100
Modify: 2019-05-21 19:13:45.884566907 +0100
Change: 2019-05-21 19:13:45.884566907 +0100
 Birth: -
ls -al /var/www/html/admin
total 276
drwxr-xr-x 7 root     root      4096 May 21 19:13 .
drwxrwxr-x 4 www-data www-data  4096 May 21 19:13 ..
-rw-r--r-- 1 root     root     11342 May 21 19:13 api_db.php
-rw-r--r-- 1 root     root      8760 May 21 19:13 api_FTL.php
-rw-r--r-- 1 root     root      3609 May 21 19:13 api.php
-rw-r--r-- 1 root     root      2795 May 21 19:13 auditlog.php
-rw-r--r-- 1 root     root      2339 May 21 19:13 CONTRIBUTING.md
-rw-r--r-- 1 root     root      2244 May 21 19:13 db_graph.php
-rw-r--r-- 1 root     root      4273 May 21 19:13 db_lists.php
-rw-r--r-- 1 root     root      5910 May 21 19:13 db_queries.php
-rw-r--r-- 1 root     root       948 May 21 19:13 debug.php
drwxr-xr-x 8 root     root      4096 Jun  4 08:00 .git
drwxr-xr-x 2 root     root      4096 May 21 19:13 .github
-rw-r--r-- 1 root     root       153 May 21 19:13 .gitignore
-rw-r--r-- 1 root     root      1229 May 21 19:13 gravity.php
-rw-r--r-- 1 root     root      9398 May 21 19:13 help.php
drwxr-xr-x 2 root     root      4096 May 21 19:13 img
-rw-r--r-- 1 root     root     10811 May 21 19:13 index.php
-rw-r--r-- 1 root     root     14164 May 21 19:13 LICENSE
-rw-r--r-- 1 root     root      3300 May 21 19:13 list.php
-rw-r--r-- 1 root     root      2843 May 21 19:13 network.php
-rw-r--r-- 1 root     root       652 May 21 19:13 .pullapprove.yml
-rw-r--r-- 1 root     root      5475 May 21 19:13 queries.php
-rw-r--r-- 1 root     root      1110 May 21 19:13 queryads.php
-rw-r--r-- 1 root     root      9422 May 21 19:13 README.md
drwxr-xr-x 4 root     root      4096 May 21 19:13 scripts
-rw-r--r-- 1 root     root     93198 May 21 19:13 settings.php
drwxr-xr-x 3 root     root      4096 May 21 19:13 style
-rw-r--r-- 1 root     root       939 May 21 19:13 taillog-FTL.php
-rw-r--r-- 1 root     root       931 May 21 19:13 taillog.php
-rw-r--r-- 1 root     root        45 May 21 19:13 .user.php.ini

In reply to deHakkelaar:

sudo tailf /var/log/lighttpd/access.log
1559557095|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559557096|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557097|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557113|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557114|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557115|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559557115|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557116|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559557116|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557117|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559557117|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557117|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559557117|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559557117|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559631791|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559631808|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559631808|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559631816|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559631816|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345
1559631817|192.168.2.8|GET /admin/ HTTP/1.1|403|345
1559631818|192.168.2.8|GET /favicon.ico HTTP/1.1|404|345

Folder/file permissions & ownership looks good (same on mine).

But above 403 & 404 errors not.
This could be a misconfigured lighttpd.
Did you make any changes to the lighttpd config files ?
Or did you install other software that might have ?

pi@noads:~ $ man dpkg
       -P, --purge package...|-a|--pending
              Purge  an  installed  or  already  removed package. This
              removes  everything,  including  conffiles.   If  -a  or
              --pending  is  given instead of a package name, then all
              packages unpacked or removed, but marked to be purged in
              file /var/lib/dpkg/status, are purged.

              Note:  some configuration files might be unknown to dpkg
              because they are created and handled separately  through
              the  configuration  scripts.  In  that  case, dpkg won't
              remove them by itself, but the package's  postrm  script
              (which  is  called  by  dpkg), has to take care of their
              removal during purge. Of course, this  only  applies  to
              files  in  system  directories,  not configuration files
              written to individual users' home directories.

              Purging of a package consists of the following steps:

              1. Remove the  package,  if  not  already  removed.  See
              --remove  for  detailed  information  about  how this is
              done.

              2. Run postrm script.

To reset everything to original state and restore the lighttpd config files, try purge the lighttpd package only (not dependencies) with below one:

sudo dpkg -P lighttpd

And reinstall (including dependencies):

sudo apt install --reinstall lighttpd

Might need to run Pi-hole repair afterwards to configure lighttpd properly:

pihole -r

1 Like

This did the trick, thank you!
Thanks to Mcat12 and jfb as well!

1 Like

That did the trick. I recently updated dietpi an that somehow messed up lighttpd. Uninstalled and ran repair et voila all good now.

Thanks for your help

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.