Error 500: Internal Server Error Error: Cannot open script file /var/www/html/admin/login.lp

Moto · October 29, 2024, 7:47am

Error 500: Internal Server Error
Error: Cannot open script file /var/www/html/admin/login.lp

Occurring randomly less frequently than once a day.

Core
    Version is v5.18.3-534-g250fbd21 (Latest: null)
    Branch is development
    Hash is 250fbd21 (Latest: 250fbd21)
Web
    Version is v5.21-995-g7cf66a51 (Latest: null)
    Branch is development
    Hash is 7cf66a51 (Latest: 7cf66a51)
FTL
    Version is vDev-3dd756d (Latest: null)
    Branch is development
    Hash is 3dd756da (Latest: 3dd756da)

sudo service pihole-FTL restart fixes it for a day or so

token (after pihole-FTL restart) NUs48DtI

ls -al /etc/pihole
total 255460
drwxr-xr-x 6 pihole pihole 4096 Oct 29 07:43 .
drwxr-xr-x 99 root root 4096 Oct 29 05:00 ..
-rw-rw---- 1 pihole pihole 65 Mar 29 2024 adlists.list
-rw-r----- 1 pihole pihole 44 Oct 29 07:43 cli_pw
drwxr-xr-x 2 pihole pihole 4096 Oct 27 14:20 config_backups
-rw-rw---- 1 pihole pihole 201 Jun 27 10:28 custom.list.bck
-rw-rw---- 1 pihole pihole 133 Mar 18 2024 custom.list.gsb
-rw-rw---- 1 pihole pihole 647 Oct 29 07:46 dhcp.leases
-rw-rw---- 1 pihole pihole 6029 Oct 19 23:21 dnsmasq.conf
-rw-rw---- 1 pihole pihole 651 Oct 27 14:20 dns-servers.conf
-rw-rw---- 1 pihole pihole 12 Sep 2 23:16 ftlbranch
-rw-rw---- 1 pihole pihole 48582656 Oct 28 10:03 gravity.db
-rw-rw---- 1 pihole pihole 50626560 May 25 23:41 gravity.db.gsb
-rw-rw---- 1 pihole pihole 48582656 Oct 27 07:19 gravity_old.db
drwxr-xr-x 2 pihole pihole 4096 Jun 27 11:00 hosts
-rw-rw---- 1 pihole pihole 408 Oct 27 14:20 install.log
-rw-rw---- 1 pihole pihole 3921873 Oct 27 14:20 list.11.gitlab.com.domains
-rw-rw---- 1 pihole pihole 81 Oct 27 14:20 list.11.gitlab.com.domains.sha1
-rw-rw---- 1 pihole pihole 10972543 Oct 27 14:20 list.13.www.github.developerdan.com.domains
-rw-rw---- 1 pihole pihole 98 Oct 27 14:20 list.13.www.github.developerdan.com.domains.sha1
-rw-rw---- 1 pihole pihole 2448254 Oct 27 14:20 list.1.raw.githubusercontent.com.domains
-rw-rw---- 1 pihole pihole 95 Oct 27 14:20 list.1.raw.githubusercontent.com.domains.sha1
-rw-rw---- 1 pihole pihole 65 Jun 23 03:54 local.list
-rw-r--r-- 1 root root 358 Aug 28 22:41 logrotate
-rw-rw---- 1 pihole pihole 3194880 Oct 27 14:20 macvendor.db
drwxr-xr-x 2 pihole pihole 4096 Mar 18 2024 migration_backup
drwxr-xr-x 2 pihole pihole 4096 Sep 13 09:58 migration_backup_v6
-rw-rw---- 1 pihole pihole 146 May 10 22:25 pihole-FTL.conf.bck
-rw-r----- 1 pihole pihole 92520448 Oct 29 07:43 pihole-FTL.db
-rw-r----- 1 pihole pihole 32768 Oct 29 07:56 pihole-FTL.db-shm
-rw-r----- 1 pihole pihole 531512 Oct 29 07:56 pihole-FTL.db-wal
-rw-rw---- 1 pihole pihole 51659 Oct 27 14:20 pihole.toml
-rw-rw---- 1 pihole pihole 560 Jun 27 08:31 setupVars.conf.old
-rw-rw---- 1 pihole pihole 737 Jun 27 11:00 tls_ca.crt
-rw-rw---- 1 pihole pihole 713 Jun 27 11:00 tls.crt
-rw-rw---- 1 pihole pihole 1001 Jun 27 11:00 tls.pem
-rw-rw---- 1 pihole pihole 366 Oct 28 12:58 versions

I have another pihole install which never errors. The 2 piholes have none overlapping DHCP scopes. This offending pihole has a pi.alert install accessing the API.

Moto · October 29, 2024, 9:09am

after update
Core
Version is v5.18.3-537-g112b9617 (Latest: null)
Branch is development
Hash is 112b9617 (Latest: 112b9617)
Web
Version is v5.21-1003-g8020179d (Latest: null)
Branch is development
Hash is 8020179d (Latest: 8020179d)
FTL
Version is vDev-3dd756d (Latest: null)
Branch is development
Hash is 3dd756da (Latest: 3dd756da)

new token: sEO7uyg7

rdwebdesign · October 29, 2024, 5:58pm

You are running Pi-hole v6.
The new version uses a different web server, but you still have the old web server (lighttpd) using port 80.

[✗] tcp:0.0.0.0:80 is in use by lighttpd
...
[✗] tcp:[::]:80 is in use by lighttpd

The v6 installation detected the port conflict and automatically changed the new web interface port to 8080.

You have 2 options:

access the web interface using the current configuration, but you need to add the port to the URL, like this: http://192.168.0.72:8080/admin or
if you are not using lighttpd, uninstall it and change Pi-hole v6 port back to 80 with:
```
pihole-FTL --config webserver.port '80,[::]:80,443s,[::]:443s'
```

Moto · October 29, 2024, 6:20pm

I am using lighthttp on both machines and have been happily accessing pihole on port 8080

DL6ER · November 5, 2024, 11:47am

Can you check if the file is really there? Anything special to this directory?
(I am thinking about something like on a USB drive, on a network share, ..., etc.)

The best I can say right now is that you may have a somehow broken filesystem if this isn't the case. Can you please run a filesystem check?

Moto · November 5, 2024, 1:25pm

Yes it is still there.
I have 2 piholes. Pihole2 does not error. Pihole4 has the intermittent error. Both are running on local flash drives in Raspberry Pis with up to date RpiOs. Pihole4 has a local install of Pi.Alert that queries the API every 5 minutes (I think).
Pihole4 has not errored for 2 days. Today I have configured pihole4's install of pi.alert to query pihole2 to see if the error is related to API access.
The filesystem permissions are identical on both systems.

deHakkelaar · November 5, 2024, 4:32pm

What does below one show?

$ file /var/www/html/admin/login.lp
/var/www/html/admin/login.lp: HTML document, ASCII text

Raspi's have a reputation of crashing when the supplied power is of poor quality.
Have a read below:

Moto · November 5, 2024, 4:49pm

I am using genuine Raspberry Pi power supplies. The Pi's are running headless without any USB devices connected.

deHakkelaar · November 5, 2024, 4:50pm

Have you checked the systemd journal on both?

EDIT: And below?

Moto · November 5, 2024, 5:11pm

journalctl shows multiple kernel: netlink: 8 bytes leftover after parsing attributes in process civetweb-worker. on whichever pi is being queried by pi.alert whatever that means.

There are no low voltage events.

$ file /var/www/html/admin/login.lp
/var/www/html/admin/login.lp: HTML document, ASCII text

Core
    Version is v5.18.3-537-g112b9617 (Latest: null)
    Branch is development
    Hash is 112b9617 (Latest: 112b9617)
Web
    Version is v5.21-1012-gc689fdc7 (Latest: null)
    Branch is development
    Hash is c689fdc7 (Latest: c689fdc7)
FTL
    Version is vDev-a5641e3 (Latest: null)
    Branch is development
    Hash is a5641e37 (Latest: a5641e37)

deHakkelaar · November 5, 2024, 5:12pm

Just checking

EDIT: Below can be useful for DL6ER:

Thats Pi-hole's own embedded web daemon.

rdwebdesign · November 5, 2024, 7:54pm

I never used Pi.alert, so I need to ask:

Are you sure Pi.alert is compatible with Pi-hole v6 API?
(Pi-hole v6 doesn't use lighttpd as web server and doesn't run PHP pages.)

Note:
Please, post a new debug token. The previous one expired.

Moto · November 5, 2024, 9:08pm

Pihole4 - pi.alert installed
JI21BI3J

Pihole2 - API now accessed by pi.alert on pihole4
kYy121Vy

Pi.Alert - GitHub - leiweibau/Pi.Alert: Scan the devices connected to your WIFI / LAN and alert you the connection of unknown devices. It also warns if a "always connected" device disconnects. In addition, it is possible to check web services for availability. For this purpose HTTP status codes and the response time of the service are evaluated.

rdwebdesign · November 5, 2024, 9:59pm

I'm still not sure why you are seeing the "Cannot open script ..." message, but I found 2 issues:

Pi.alert is trying to access the web interface, but it is failing to authenticate.
There are many log entries like:
```
[...] Authentication required, redirecting to /admin/login
```
Pi.alert tries to authenticate using HTTP headers, but apparently is failing.
Are you using the password to authenticate?
Are you using the IP on the URL? or hostname/domain?
The URL used to access the API is using a negative value for max_devices, but this value cannot be negative:
```
(...) WARNING: Cannot parse unsigned integer parameter max_devices in query string "max_devices=-1&max_addresses=2": Specified integer negative, this is not allowed
```
This is the Pi.alert code (line 1079) requesting the invalid value:

github.com

leiweibau/Pi.Alert/blob/32d2a194aa003851f8fe4f83d47ebbc610e4b862/back/pialert.py#L1077-L1079


      
          #max_devices=-1 seems to be "all". no more detailed information found in the API documentation
          #max_addresses=2 IPs per host
          raw_deviceslist = requests.get(PIHOLE6_URL+'api/network/devices?max_devices=-1&max_addresses=2', headers=headers, verify=False)

Moto · November 5, 2024, 10:55pm

I am using the password and the configured URL uses its IP address

Moto · November 6, 2024, 8:45am

I have set the max_devices to 30 - 6 more than the number of devices on my network. Is there a value to capture all clients.
I have now reverted pi.alert to query localhost on pihole4 to see if the server error returns.

DL6ER · November 6, 2024, 9:37am

It means nothing, it's a mere warning that we have sent a too large netlink packet to the kernel. It won't have any consequences. This harmless warning will be fixed by:

github.com/pi-hole/FTL

Fix netlink message length

pi-hole:development ← pi-hole:fix/netlink_msg_size

opened 09:37AM - 06 Nov 24 UTC

DL6ER

+8 -2

# What does this implement/fix? Increase buffer size of netlink messages to 3…2 KB and fix incorrect nl message length when requesting link information. Fixes kernel warnings reported [on Discourse](https://discourse.pi-hole.net/t/error-500-internal-server-error-error-cannot-open-script-file-var-www-html-admin-login-lp/73698/10). ### How to reproduce 1. Run `sudo dmesg --clear` 2. Go to https://pi.hole/admin/interfaces 3. Run `sudo dmesg --read-clear` 4. See (possibly many) entries like ``` [2290187.418532] netlink: 8 bytes leftover after parsing attributes in process `civetweb-worker'. [2290187.418538] netlink: 8 bytes leftover after parsing attributes in process `civetweb-worker'. ``` (the number of entries depends on the number of interfaces on your system) 5. Switch to this branch (`fix/netlink_msg_size`), repeat steps 1. - 4. and confirm that these message are now absent --- **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

Moto · November 12, 2024, 9:57pm

The error 500 is still occurring. Token before pihole-FTL restarted 8ppqhJ0X

DL6ER · November 13, 2024, 1:31pm

There are several errors in the Tricorder log related to an no file descriptors being available on your system. This is an OS-level error indicating that you have too many files open at the same time. I see issues related to the database, for connecting with other processes (e.g. ip) and it would be quite likely that this is also the cause of your 500 errors - when FTL cannot open the file for reading, an error 500 is the natural consequence.

Please check the number of allowed open files at the same time ( ulimit -n ) on your system.

Also, please run sudo lsof | grep pihole and provide the output

Moto · November 13, 2024, 5:24pm

ulimit -n
1024

sudo lsof | grep pihole - there are over 16000 entries in the result!
I rebooted and reran the command and got 650 entries