read chunked responses ... doesn't seem to work in chunks

[jonathon-love]

hi,

i'm reading chunked responses from a requests.post(...), however calling local chunk, err = r:iter_content() doesn't give me a chunk each time, but rather many of the chunks all joined together. i.e. the following outputs a single big chunk.

local chunk, err = r:iter_content()

ngx.log(ngx.ERR, "CHUNK")
ngx.log(ngx.ERR, chunk)
ngx.log(ngx.ERR, "END CHUNK")

if i access the same endpoint with, say, a js XHR i successfully get each of the chunks coming through separately.

if i set a very small chunk size in the r:iter_content() call, i can get an approximation of the desired behaviour (indicating that it is receiving the chunks from the server in a chunk like fashion ... and i can also confirm that the chunked encoding heading is being successfully read in the r.headers.

i am using the 7.3-1 from luarocks.

any tips on how i could debug this?

with thanks

[tokers]

Hi,

When handling the chunked response. r:iter_content doesn't return content according to the chunk boundary. Instead, it tries to read data until the size is equal to the one you want (pass from the size arg in the iter_content method or using the default one), unless the rest content is less than the one you specified.

If you want to read each chunk body one by one, we may have to update the implementation of the r:iter_content().

[jonathon-love]

hi,

thanks for clarifying.

we stream progress information (jsonlines) back from our service, so we want to be able to flush each chunk back to the client. for now i've worked around it as follows:

local prev

while true do
    local chunk, err = r:iter_content(100)

    if not chunk then
        ngx.log(ngx.ERR, err)
        return
    end

    if chunk == "" then
        break
    end

    if prev then
        chunk = prev .. chunk
    end

    local last
    _, _, last = string.find(chunk, "([^\n]+)\n$")
    if last then
        prev = nil
    else
        _, _, last, prev = string.find(chunk, "([^\n]+)\n([^\n]+)$")
    end

    if last then
        ngx.print(last .. "\n")
        ngx.flush()
    else
        prev = chunk
    end
end

if prev then
    ngx.print(prev)
end

but it would be nice if we could just do:

while true do
    local chunk, err = r:iter_content(100)

    if not chunk then
        ngx.log(ngx.ERR, err)
        return
    end

    if chunk == "" then
        break
    end

    ngx.print(chunk)
    ngx.flush()

:)