This is causing issues for me with the orbital-sync library, but I suspect it's also causing issues for me with uploading the teleporter zip files - they fail to upload about 90% of the time, sometimes randomly succeeding.
I suspect it's to do with the server not supporting the transport padding, but I need to do some more testing.
No - doesn't look like transport padding - seems to just not work.
teleporter-upload.pcapng.zip (24.0 KB)
filter with ip.dst == 192.168.0.98 and tcp[80] and http.request.method == POST - none have got transport padding between the ending boundary -- and the CRLF.
Here's the upload I'm using BTW:
pi-hole_auk_teleporter_2024-08-05_19-58-24_BST.zip (19.1 KB)
The expected encoding is indeed multipart/form-data for your payload as specified in our OpenAPI specs. I will try to find some time tomorrow to test wit your particular file. However, I use Teleporter nearly on a daily basis during v6 development and haven't had a single issue. Do you also see similar uploading issues when using the web interface? If not, what is the difference?
edit Just in case this is useful for you, this is how my Python script does it with 100% success rate:
with requests.post(url = self.api_url + "/api/teleporter", files = {"file": ('teleporter.zip', teleporter_archive, 'application/zip')}, headers=self.headers) as response:
return response.json()
where teleporter_archive is the ZIP archive in memory (bytes).
Web interface is fine. See the linked GitHub issue on civitweb - browsers (or at least Chrome / Edge) add CRLF to the end of the last boundary, whereas node-fetch doesn't.
The RFC for multipart/form-data states that the CRLF is optional. RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types (rfc-editor.org) - specifically the BNF:
multipart-body := [preamble CRLF]
dash-boundary transport-padding CRLF
body-part *encapsulation
close-delimiter transport-padding
[CRLF epilogue]
Python requests is working for you because it also includes the optional trailing CRLF (0d0a)
root@auk:/# tcpflow -D -c -i lo port 80
reportfilename: ./report.xml
tcpflow: listening on lo
192.168.000.098.37068-192.168.000.098.00080:
0000: 504f 5354 202f 6170 692f 7465 6c65 706f 7274 6572 2048 5454 502f 312e 310d 0a48 POST /api/teleporter HTTP/1.1..H
0020: 6f73 743a 2031 3932 2e31 3638 2e30 2e39 380d 0a55 7365 722d 4167 656e 743a 2070 ost: 192.168.0.98..User-Agent: p
0040: 7974 686f 6e2d 7265 7175 6573 7473 2f32 2e32 382e 310d 0a41 6363 6570 742d 456e ython-requests/2.28.1..Accept-En
0060: 636f 6469 6e67 3a20 677a 6970 2c20 6465 666c 6174 650d 0a41 6363 6570 743a 202a coding: gzip, deflate..Accept: *
0080: 2f2a 0d0a 436f 6e6e 6563 7469 6f6e 3a20 6b65 6570 2d61 6c69 7665 0d0a 436f 6e74 /*..Connection: keep-alive..Cont
00a0: 656e 742d 4c65 6e67 7468 3a20 3138 310d 0a43 6f6e 7465 6e74 2d54 7970 653a 206d ent-Length: 181..Content-Type: m
00c0: 756c 7469 7061 7274 2f66 6f72 6d2d 6461 7461 3b20 626f 756e 6461 7279 3d38 3662 ultipart/form-data; boundary=86b
00e0: 3936 3565 3534 6234 3362 6233 6530 3164 6437 3433 6337 3333 3635 3636 370d 0a0d 965e54b43bb3e01dd743c73365667...
0100: 0a .
192.168.000.098.37068-192.168.000.098.00080:
0000: 2d2d 3836 6239 3635 6535 3462 3433 6262 3365 3031 6464 3734 3363 3733 3336 3536 --86b965e54b43bb3e01dd743c733656
0020: 3637 0d0a 436f 6e74 656e 742d 4469 7370 6f73 6974 696f 6e3a 2066 6f72 6d2d 6461 67..Content-Disposition: form-da
0040: 7461 3b20 6e61 6d65 3d22 6669 6c65 223b 2066 696c 656e 616d 653d 2274 656c 6570 ta; name="file"; filename="telep
0060: 6f72 7465 722e 7a69 7022 0d0a 436f 6e74 656e 742d 5479 7065 3a20 6170 706c 6963 orter.zip"..Content-Type: applic
0080: 6174 696f 6e2f 7a69 700d 0a0d 0a0d 0a2d 2d38 3662 3936 3565 3534 6234 3362 6233 ation/zip......--86b965e54b43bb3
00a0: 6530 3164 6437 3433 6337 3333 3635 3636 372d 2d0d 0a e01dd743c73365667--..
I've started work on a patch for civitweb - Comparing civetweb:master...timtjtim:multipart-form-data-#1276 · civetweb/civetweb (github.com).
And sorry - yes I do get some issues using the web interface, but I think there's a race condition in some file creation going on, I'm still investigating that. The web interface issues (for me) are unrelated to this HTTP issue
Multipart form data #1276 by timtjtim · Pull Request #1277 · civetweb/civetweb (github.com)
I've opened a PR for civitweb.
@DL6ER what's the process for updating the embedded civitweb within Pi Hole?
@DL6ER that PR is now merged into civitweb
Sorry, life's been quite hectic the last few days. The procedure is: once they push a release, we import it. We have a couple of patches against the last release which I always wanted to submit against CivetWeb. Not all of them are applicable for general audience, so some will remain in our realm. There is also a rather experimental fix
but it is not a complete fix - it only fixes Lua server pages (the only ones really relevant to Pi-hole). It fixes 404 Pages (sometimes?) respond with a 200 HTTP status. I may submit it partially and leave the rest to them, it's pretty straightforward. In an ideal world, I should have time later today but ... let's say: wel'll see.
Once we have also put the Pi-hole patches in, we can have a look how/when to update the embedded CivetWeb. IMO it'd be cleaner to wait for a release, could maybe argue that it's time for another one. If they refuse this for incomprehensible reasons, we can think about migrating to their master head.
Done, see my comment here:
@timtjtim You can go ahead and try the new FTL branch update/civetweb using
pihole checkout ftl update/civetweb
which - you already guessed from its name - updates CivetWeb inside FTL to the lastest master tip of the upstream project. You may have noticed that this triggered four PRs in the CivetWeb repository so it was worth it also for them. The last one changes something I have only noticed because of your recent changes to the forms code.
That last one is a nice change, gets rid of a lot of casting, plus the warnings - good spot 
I'll have a play around today / tomorrow.
The change has just been merged into develpoment-v6