Repository navigation
Closed
Description
Hello everyone, thanks for maintaining rsync 👋🏻 🥳
We're using rsync-ssl in Arch Linux for mirror synchronization and when upgrading our server fleet we noticed that all synchronizations failed with the following error when upgrading openssl to 3.6.2:
Apr 11 13:09:07 mirror.pkgbuild.com systemd[1]: Starting Synchronize repo mirror...
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7055]: verify depth is 4
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7055]: Connecting to 2a01:4f8:251:598::
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: receiving file list ... done
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7057]: rsync: connection unexpectedly closed (4154299 bytes received so far) [receiver]
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7057]: rsync error: error in rsync protocol data stream (code 12) at io.c(232) [receiver=3.4.1]
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: rsync: [generator] write error: Broken pipe (32)
Apr 11 13:09:08 mirror.pkgbuild.com syncrepo[7054]: rsync error: error in socket IO (code 10) at io.c(849) [generator=3.4.1]
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: syncrepo.service: Main process exited, code=exited, status=10/n/a
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: syncrepo.service: Failed with result 'exit-code'.
Apr 11 13:09:08 mirror.pkgbuild.com systemd[1]: Failed to start Synchronize repo mirror.
When downgrading back to 3.6.1 everything worked again. Any idea how we could start to debug this in more detail? Adding any verbosity flags didn't show more useful information. 🤔
The script that we're using rsync with is the following:
#!/bin/bash
target="/srv/ftp"
lock="/run/lock/syncrepo.lck"
source_url='rsync://rsync.archlinux.org/ftp_tier1'
lastupdate_url='https://rsync.archlinux.org/lastupdate'
[ ! -d "${target}" ] && mkdir -p "${target}"
exec 9>"${lock}"
flock -n 9 || exit
rsync_cmd() {
local -a cmd=(rsync-ssl --type=openssl --verbose -rlptH --safe-links --delete-delay --delay-updates
"--timeout=600" --no-motd)
if stty &>/dev/null; then
cmd+=(-h -v --progress)
else
cmd+=("--info=name1")
fi
"${cmd[@]}" "$@"
}
# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && [[ -f "$target/lastupdate" ]] && diff -b <(curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
# keep lastsync file in sync for statistics generated by the Arch Linux website
rsync_cmd "$source_url/lastsync" "$target/lastsync"
exit 0
fi
rsync_cmd \
--exclude="/other" \
--exclude="/sources" \
--exclude="*-debug/" \
"${source_url}" "${target}"If you're interested, here is the incident in detail: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/818
Reacted by Leo and Leonidas Spyropoulos
Metadata
Metadata
Assignees
Labels
No labels
Activity
zyxhere commentedon Apr 17, 2026
Wouldn't this be a bug in OpenSSL? A minor patch update broke one of its consumer.
Not saying that Rsync isn't at fault here (it very well could be) just think this bug should go to OpenSSL first.
devZer0 commentedon Apr 20, 2026
@christian-heusel , i'm trying to get a clue how to debug this, and while reading into this, chatgpt spits me out this one:
not sure if this is all true (as we know AI talks a lot of shit, too), but it's often valuable infos,
so, maybe it's worth a try using rsync-ssl in stunnel mode and have a look, how that behaves after openssl update?
stunnel should use openssl shared libs and if there really is an issue at ssl level (and not an s_client issue) , you may see simiilar problem there. and if so, it should be easier to debug , as you could add debug and logfile options to stunnel command in rsync-ssl
devZer0 commentedon Apr 25, 2026
i build a reprod-case (stunnel in front of rsyncd) and for me it works without a problem when using rsync-ssl in stunnel mode.
with rsync-ssl in s_client/openssl mode, i can reproduces your issue and my transfer session simply bails out after 3/4 of the transfer like this:
meanwhile i have found this one
openssl/openssl#30925
s_client silently drops session on a BUFSIZZ sized stdin read (regression in 3.6.2) #30925
anyhow, researching on this with chatgpt , i'm getting constantly nagged, that s_client would be no realiable means for data transport, but only a tool meant for debugging/testing. anthropic claude also confirms this.
in other words
given the fact that we have an issue here with s_client which does not seem to easily be resolved - i tried another option i remembered using quite a while ago.
there is network swiss army knife and netcat on steroids tool named "socat".
i put this line in rsync-ssl as quick and dirty thest (abusing stunnel mode) and it also works well
so, the question is if we shouldn't add socat as a transport option and put some energy into searching for reliable information if it's good to keep s_client as an transport option at all - or better drop it entirely, if stunnel and socat are valid replacements / alernatives
@christian-heusel @WayneD ,what do you think ?
i also made some interesting findings regarding the issue with gnutls-cli, which also seems to be related to buffers - see https://gitlab.com/gnutls/gnutls/-/work_items/1037
devZer0 commentedon Apr 27, 2026
just having tested the fix at openssl/openssl#30925 and it works
christian-heusel commentedon Apr 27, 2026
Nice, thanks for confirming and all the investigation around it! 🤗 I have followed the upstream bug in order to be notified about any progress there!
Skip OpenSSL upgrade to 3.6.2 for archive.archlinux.org
pfactum commentedon May 8, 2026
Simplifying a workaround with
socatmentioned earlier by @devZer0, it can be possible to avoid using thersync-sslwrapper altogether and just specify the--rshoption forrsyncdirectly like this:$ rsync --rsh="sh -c 'exec socat - OPENSSL:example.com:874' --" …devZer0 commentedon May 8, 2026
so, if we can do it that way, do we need rsync-ssl at all?
couldn't we use s_client, gnutls-cli, socat, stunnel and ncat all this way @pfactum ?
will dig into this when i find time for it.
anyhow, it's weird that nobody seems to care for the regression in openssl. maybe they have too much to do fixing security issues because so many being found with help of AI....
pfactum commentedon May 8, 2026
rsync-sslis just a nice wrapper around all of this, but its function essentially boils down to what can be represented by a single--rsh='…'argument.steadytao commentedon Jun 6, 2026
Thank you all for your investigation and work on this.
devZer0 commentedon Jun 6, 2026
i can't yet confirm that this is fixed, i built openssl from git/master today and tried that and with openssl s_client still bails out with
@christian-heusel , maybe you also want to test?
pfactum commentedon Jun 6, 2026
@steadytao Would you be able point out the exact fix for this please? Thank you.
steadytao commentedon Jun 6, 2026
Reopening this; I read this wrongly and closed too early. Thank you for catching me up on this.
The investigation points strongly at the OpenSSL
s_clientregression tracked in openssl/openssl#30925 but that issue is still open and there does not appear to be a confirmed exact fix yet. The stunnel/socat paths look like useful workarounds while awaiting that but I do not want the closed state here to imply the rsync-ssl/OpenSSL path is confirmed fixed.steadytao commentedon Jun 6, 2026
I think the rsync-side fix here is to add
socatas an explicitrsync-ssl --type=socattransport rather than trying to work around OpenSSLs_clientbuffering inside rsync while we await that upstream. That keeps the existing OpenSSL path for compatibility, gives users a transport that has already worked around this failure mode and avoids making a default-order change, at least for now.devZer0 commentedon Jun 6, 2026
thanks. i have working prototype of rsync-ssl with socat and ncat as fourth and fifth option for transport, but it's not yet in good shape.
steadytao commentedon Jun 6, 2026
I opened the above PR with roughly the same shape I stated earlier.
It adds
rsync-ssl --type=socatand keeps the existing OpenSSL path for compatibility. The default search still prefers OpenSSL first but if OpenSSL is unavailable it now tries socat before stunnel because socat can still verify with the default CA behaviour while this script's stunnel path cannot.I kept
ncatout of this PR so the change stays focused; it can be added separately if we want another transport option.mattcaswell commentedon Jun 8, 2026
OpenSSL fix is here:
openssl/openssl@037ed10
mattcaswell commentedon Jun 8, 2026
New OpenSSL releases containing the above fix are due out tomorrow.
steadytao commentedon Jun 8, 2026
Thank you for the notice!
devZer0 commentedon Jun 9, 2026
i can confirm the problem is fixed
devZer0 commentedon Jun 9, 2026
@christian-heusel , can you also confirm and/or close this ticket ?
steadytao commentedon Jun 9, 2026
socat support is still useful so I will leave the PR open but the issue itself is patched and I have confirmed such. Thanks for your assistance everyone.
Stop ignoring openssl upgrade
Stop ignoring openssl upgrade
Antiz96 commentedon Jun 14, 2026
As a member of the Arch Linux DevOps team as well, I'm allowing myself to answer for Christian: yes, we can confirm the issue is fixed (even though it has already been confirmed by multiple people in this issue). Thanks a lot 🤗