Multiple Archive URL for mirrors

[AlexeiKlimenko]

Good day.

Sometimes (and very often) during aptly mirror update processing we're facing with various issues such as EOF, timeout, unexpected length etc. It leads that updating of mirror is gonna to be failed.
Have you got any ideas/suggestions how to define multiple URL for Archive URL as a endpoints for balancing? Mean, than we face again with issue try to use another URL for this mirror (another backend)?

[neolynx]

what version of aptly are you using?

for unstable connections, the grab downloader is recommended, and you can set the download retry attempts. what downloader is configured in your aptly.conf ?

[AlexeiKlimenko]

aptly version: 1.4.0

Part of config:
"downloadConcurrency": 200, "downloadSpeedLimit": 0, "downloadRetries": 0, "databaseOpenAttempts": -1,

We'd prefer to use multiple endpoints instead of specific one and looking for the way how to implement it.

[neolynx]

I see your point for having multiple/backup URLs, but implementing this would require a lot of cross checking (Release, Package files, signatures).. so I think this would be difficult.

However, we solved such kind of problems by using the grab downloader, which is more robust in retrying and continuing downloading.

could you check if the behavior improved by setting downloader: "grab" and "downloadRetries": 10 ?

[AlexeiKlimenko]

Our current version aptly (version: 1.4.0) knows nothing about this type of downloader. We've implemented nginx for using upstreams for mirrors and redefined archive URL referring to nginx. After that every time receive this error on any mirror:
STDOUT: ERROR: unable to update: could not determine length of http://xxx/debian/dists/bookworm/main/installer-arm64/current/images/netboot/debian-installer/arm64/grub/arm64-efi/fdt.lst

[neolynx]

Aptly 1.6.0 has been releassed: #1414

Could you try the grab downloader with this version ?

I did not fully understand how you implemented the nginx proxy... is this related to mirror downloads, i.e. caching aptly requests ?
for caching apt repositories, I had better experience with a dedicated apt cacher. I would suggest to try aptly 1.6.0 with direct downloads, and grab configured with retries.

if the error persists, please share some aptly logs to analyze...