Skip to content

rsync-ssl breaking with openssl 3.6.2 #869

Closed

Description

@christian-heusel

Hello everyone, thanks for maintaining rsync 👋🏻 🥳

We're using rsync-ssl in Arch Linux for mirror synchronization and when upgrading our server fleet we noticed that all synchronizations failed with the following error when upgrading openssl to 3.6.2:

Apr 11 13:09:07 mirror.pkgbuild.com systemd[1]: Starting Synchronize repo mirror...
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7055]: verify depth is 4
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7055]: Connecting to 2a01:4f8:251:598::
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: receiving file list ... done
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7057]: rsync: connection unexpectedly closed (4154299 bytes received so far) [receiver]
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7057]: rsync error: error in rsync protocol data stream (code 12) at io.c(232) [receiver=3.4.1]
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: rsync: [generator] write error: Broken pipe (32)
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: rsync error: error in socket IO (code 10) at io.c(849) [generator=3.4.1]
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: syncrepo.service: Main process exited, code=exited, status=10/n/a
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: syncrepo.service: Failed with result 'exit-code'.
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: Failed to start Synchronize repo mirror.

When downgrading back to 3.6.1 everything worked again. Any idea how we could start to debug this in more detail? Adding any verbosity flags didn't show more useful information. 🤔

The script that we're using rsync with is the following:

#!/bin/bash

target="/srv/ftp"
lock="/run/lock/syncrepo.lck"
source_url='rsync://rsync.archlinux.org/ftp_tier1'
lastupdate_url='https://rsync.archlinux.org/lastupdate'

[ ! -d "${target}" ] && mkdir -p "${target}"

exec 9>"${lock}"
flock -n 9 || exit

rsync_cmd() {
        local -a cmd=(rsync-ssl --type=openssl --verbose -rlptH --safe-links --delete-delay --delay-updates
                "--timeout=600" --no-motd)

        if stty &>/dev/null; then
                cmd+=(-h -v --progress)
        else
                cmd+=("--info=name1")
        fi

        "${cmd[@]}" "$@"
}

# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
        # keep lastsync file in sync for statistics generated by the Arch Linux website
        rsync_cmd "$source_url/lastsync" "$target/lastsync"
        exit 0
fi

rsync_cmd \
        --exclude="/other" \
        --exclude="/sources" \
        --exclude="*-debug/" \
        "${source_url}" "${target}"

If you're interested, here is the incident in detail: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/818

cc @klausenbusk @eworm-de @jelle

Activity

  1. zyxhere commented on Apr 17, 2026

    @zyxhere

    Wouldn't this be a bug in OpenSSL? A minor patch update broke one of its consumer.
    Not saying that Rsync isn't at fault here (it very well could be) just think this bug should go to OpenSSL first.

  2. devZer0 commented on Apr 20, 2026

    @devZer0

    @christian-heusel , i'm trying to get a clue how to debug this, and while reading into this, chatgpt spits me out this one:

    • s_client is not designed as a transport layer
    • even small output breaks rsync
    • TLS renegotiation / EOF handling can hang

    not sure if this is all true (as we know AI talks a lot of shit, too), but it's often valuable infos,

    so, maybe it's worth a try using rsync-ssl in stunnel mode and have a look, how that behaves after openssl update?

    stunnel should use openssl shared libs and if there really is an issue at ssl level (and not an s_client issue) , you may see simiilar problem there. and if so, it should be easier to debug , as you could add debug and logfile options to stunnel command in rsync-ssl

  3. devZer0 commented on Apr 25, 2026

    @devZer0

    i build a reprod-case (stunnel in front of rsyncd) and for me it works without a problem when using rsync-ssl in stunnel mode.

    with rsync-ssl in s_client/openssl mode, i can reproduces your issue and my transfer session simply bails out after 3/4 of the transfer like this:

    include/clocksource/timer-goldfish.h
    include/clocksource/timer-riscv.h
    include/clocksource/timer-ti-dm.h
    include/clocksource/timer-xilinx.h
    rsync: connection unexpectedly closed (4754159 bytes received so far) [receiver]
    rsync error: error in rsync protocol data stream (code 12) at io.c(232) [receiver=3.4.1]
    rsync: connection unexpectedly closed (4330075 bytes received so far) [generator]
    rsync error: error in rsync protocol data stream (code 12) at io.c(232) [generator=3.4.1]
    

    meanwhile i have found this one

    openssl/openssl#30925
    s_client silently drops session on a BUFSIZZ sized stdin read (regression in 3.6.2) #30925

    anyhow, researching on this with chatgpt , i'm getting constantly nagged, that s_client would be no realiable means for data transport, but only a tool meant for debugging/testing. anthropic claude also confirms this.

    in other words

    • the key takeaway is: 👉 OpenSSL 3.6 tightened / exposed edge cases around partial writes
    • OpenSSL 3.x generally: relies more on internal buffering / BIO abstraction , has stricter handling of non-blocking / partial writes - This means: if your program pipes data into s_client or you use it interactively, you may now hit short write behavior that older versions masked
    • When rsync-ssl uses openssl s_client as a transport, it assumes a fairly “stream-like” pipe. With OpenSSL 3.x, s_client is a bit less forgiving: partial writes and buffering are exposed instead of silently retried, which can break tools that expect blocking, full writes.

    given the fact that we have an issue here with s_client which does not seem to easily be resolved - i tried another option i remembered using quite a while ago.

    there is network swiss army knife and netcat on steroids tool named "socat".

    i put this line in rsync-ssl as quick and dirty thest (abusing stunnel mode) and it also works well

            # devzero@web.de came up with this no-tmpfile calling syntax:
    #       exec $RSYNC_SSL_STUNNEL -fd 10 11<&0 <<EOF 10<&0 0<&11 11<&-
            exec socat - OPENSSL:$hostname:$port,verify=0
    

    so, the question is if we shouldn't add socat as a transport option and put some energy into searching for reliable information if it's good to keep s_client as an transport option at all - or better drop it entirely, if stunnel and socat are valid replacements / alernatives

    @christian-heusel @WayneD ,what do you think ?

    i also made some interesting findings regarding the issue with gnutls-cli, which also seems to be related to buffers - see https://gitlab.com/gnutls/gnutls/-/work_items/1037

  4. devZer0 commented on Apr 27, 2026

    @devZer0

    just having tested the fix at openssl/openssl#30925 and it works

  5. christian-heusel commented on Apr 27, 2026

    @christian-heusel
    Author

    Nice, thanks for confirming and all the investigation around it! 🤗 I have followed the upstream bug in order to be notified about any progress there!

  6. pfactum commented on May 8, 2026

    @pfactum

    Simplifying a workaround with socat mentioned earlier by @devZer0, it can be possible to avoid using the rsync-ssl wrapper altogether and just specify the --rsh option for rsync directly like this:

    $ rsync --rsh="sh -c 'exec socat - OPENSSL:example.com:874' --" …
  7. devZer0 commented on May 8, 2026

    @devZer0

    so, if we can do it that way, do we need rsync-ssl at all?

    couldn't we use s_client, gnutls-cli, socat, stunnel and ncat all this way @pfactum ?

    will dig into this when i find time for it.

    anyhow, it's weird that nobody seems to care for the regression in openssl. maybe they have too much to do fixing security issues because so many being found with help of AI....

  8. pfactum commented on May 8, 2026

    @pfactum

    rsync-ssl is just a nice wrapper around all of this, but its function essentially boils down to what can be represented by a single --rsh='…' argument.

  9. steadytao commented on Jun 6, 2026

    @steadytao
    Member

    Thank you all for your investigation and work on this.

  10. devZer0 commented on Jun 6, 2026

    @devZer0

    i can't yet confirm that this is fixed, i built openssl from git/master today and tried that and with openssl s_client still bails out with

    /usr/bin/rsync-ssl --type=openssl --port 8873 -av --dry-run debian::test /tmp
    <snipp>
    include/clocksource/timer-riscv.h
    include/clocksource/timer-ti-dm.h
    include/clocksource/timer-xilinx.h
    rsync: connection unexpectedly closed (4754167 bytes received so far) [receiver]
    rsync error: error in rsync protocol data stream (code 12) at io.c(232) [receiver=3.4.1]
    rsync: connection unexpectedly closed (4330047 bytes received so far) [generator]
    rsync error: error in rsync protocol data stream (code 12) at io.c(232) [generator=3.4.1]
    

    @christian-heusel , maybe you also want to test?

  11. pfactum commented on Jun 6, 2026

    @pfactum

    @steadytao Would you be able point out the exact fix for this please? Thank you.

  12. steadytao commented on Jun 6, 2026

    @steadytao
    Member

    Reopening this; I read this wrongly and closed too early. Thank you for catching me up on this.

    The investigation points strongly at the OpenSSL s_client regression tracked in openssl/openssl#30925 but that issue is still open and there does not appear to be a confirmed exact fix yet. The stunnel/socat paths look like useful workarounds while awaiting that but I do not want the closed state here to imply the rsync-ssl/OpenSSL path is confirmed fixed.

  13. steadytao commented on Jun 6, 2026

    @steadytao
    Member

    I think the rsync-side fix here is to add socat as an explicit rsync-ssl --type=socat transport rather than trying to work around OpenSSL s_client buffering inside rsync while we await that upstream. That keeps the existing OpenSSL path for compatibility, gives users a transport that has already worked around this failure mode and avoids making a default-order change, at least for now.

  14. devZer0 commented on Jun 6, 2026

    @devZer0

    thanks. i have working prototype of rsync-ssl with socat and ncat as fourth and fifth option for transport, but it's not yet in good shape.

  15. steadytao commented on Jun 6, 2026

    @steadytao
    Member

    I opened the above PR with roughly the same shape I stated earlier.

    It adds rsync-ssl --type=socat and keeps the existing OpenSSL path for compatibility. The default search still prefers OpenSSL first but if OpenSSL is unavailable it now tries socat before stunnel because socat can still verify with the default CA behaviour while this script's stunnel path cannot.

    I kept ncat out of this PR so the change stays focused; it can be added separately if we want another transport option.

  16. mattcaswell commented on Jun 8, 2026

    @mattcaswell

    OpenSSL fix is here:

    openssl/openssl@037ed10

  17. mattcaswell commented on Jun 8, 2026

    @mattcaswell

    New OpenSSL releases containing the above fix are due out tomorrow.

  18. steadytao commented on Jun 8, 2026

    @steadytao
    Member

    New OpenSSL releases containing the above fix are due out tomorrow.

    Thank you for the notice!

  19. devZer0 commented on Jun 9, 2026

    @devZer0

    i can confirm the problem is fixed

  20. devZer0 commented on Jun 9, 2026

    @devZer0

    @christian-heusel , can you also confirm and/or close this ticket ?

  21. steadytao commented on Jun 9, 2026

    @steadytao
    Member

    socat support is still useful so I will leave the PR open but the issue itself is patched and I have confirmed such. Thanks for your assistance everyone.

  22. Antiz96 commented on Jun 14, 2026

    @Antiz96

    @christian-heusel , can you also confirm and/or close this ticket ?

    As a member of the Arch Linux DevOps team as well, I'm allowing myself to answer for Christian: yes, we can confirm the issue is fixed (even though it has already been confirmed by multiple people in this issue). Thanks a lot 🤗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions