Skip to content

Stream HTTP request bodies for large file uploads - #663

Merged
bsergean merged 1 commit into
masterfrom
feature/http-upload-stream
Nov 18, 2026
Merged

bsergean merged 1 commit into
masterfrom
feature/http-upload-stream

Conversation

@bsergean

Copy link
Copy Markdown
Collaborator

The HTTP client currently requires the whole upload body in a string and copies it into the serialized request. Add a known-length streaming body so applications can upload files larger than 2 GiB with bounded client buffering.

  • Set HttpRequestArgs::bodySize (uint64_t) and onUploadCallback(offset, buffer, capacity), then pass an empty string body to POST/PUT/PATCH or request. The callback fills a reusable buffer of at most 64 KiB; requests use ordinary Content-Length framing.
  • Preserve the source and length in asynchronous request snapshots, replay from offset zero on redirects, and check cancellation/transfer deadlines between chunks. Short reads are supported; premature EOF, invalid byte counts and callback exceptions fail the request.
  • Reject conflicting string bodies, request compression and explicit framing headers. Existing string-based uploads retain their behavior. Include a file-upload example and document source ownership and replay requirements.

This addresses client-side uploads in #556. Chunked request encoding (#381) is separate and unnecessary when the body length is known. Server-side request parsing and the existing multipart form helpers still buffer their complete bodies.

Validation:

  • Full local suite with TLS disabled: 33/33 executables pass.
  • OpenSSL upload/async/response tests pass. Real HTTP and HTTPS POST/PUT/PATCH uploads and redirects are covered; the upload executable passes 99 assertions in 5 cases, with individual cases taking about 0–10 ms locally.
  • A declared length above 4 GiB is checked without allocating or transmitting gigabytes. Tests also cover short reads, binary data, cancellation, failed sources, empty bodies, framing conflicts and queued configuration snapshots.
  • C++11 production builds with -pedantic-errors, formatting and git diff --check pass.

Closes #556.

@bsergean
bsergean merged commit 15a06b8 into master Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to upload large file?

1 participant