Repository navigation
AWS S3 Backup fails with SSL_CTX_load_verify_locations #3010
Description
I configured automatic backups with storing the backup archive in AWS S3 (self hosted minio cluster). After upgrade the docker image from 17.1.1 to 17.3.2 the backup configuration fails with a SSL error SSL_CTX_load_verify_locations. The configuration is not changed. Server certificate of minio cluster is a Let's Encrypt certificate.
Configuration:
GITLAB_BACKUP_SCHEDULE: daily
GITLAB_BACKUP_TIME: "23:00"
GITLAB_BACKUP_SKIP: "uploads,artifacts,packages,builds,lfs,registry,ci_secure_files,terraform_state"
GITLAB_BACKUP_EXPIRY: 302400
AWS_BACKUPS: "true"
AWS_BACKUP_REGION: "de-local-1"
AWS_BACKUP_BUCKET: "gitlab-backup"
AWS_BACKUP_ACCESS_KEY_ID: <masked>
AWS_BACKUP_SECRET_ACCESS_KEY: <masked>
AWS_BACKUP_ENDPOINT: https://main.storage.<domain>
Log extract:
2024-09-23 23:00:33 +0200 -- Creating backup archive: 1727125223_2024_09_23_17.3.2_gitlab_backup.tar ...
rake aborted!
Excon::Error::Socket: SSL_CTX_load_verify_locations: system lib (OpenSSL::SSL::SSLError)
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `socket'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:121:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/mock.rb:57:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:286:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:677:in `_request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:672:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:324:in `multipart_save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:279:in `save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/home/git/gitlab/lib/backup/remote_storage.rb:26:in `upload'
/home/git/gitlab/lib/backup/manager.rb:279:in `upload'
/home/git/gitlab/lib/backup/manager.rb:151:in `run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:32:in `create'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:75:in `lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:117:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Activity
ymazzer commentedon Sep 27, 2024
Same here, thought it was linked to openssl3 upcoming requirements.
Any idea
dimm0 commentedon Sep 30, 2024
Same here with google s3
ntnamazu commentedon Sep 30, 2024
I have seen the same error message and a workaround like below to add symbolic link was posted at GitLab Forum. The post said that "Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there". This method quite worked for me.
Also, I implemented this workaround in PR #3016 as commit d8d7804, so if the PR is approved and merged, I think this issue would be resolved.
ymazzer commentedon Oct 1, 2024
Just tested, works well. Thanks for the tip!
Btw, would have been better to keep the fix in another PR.
fix(sameersbn#3010) Add symbolic link to the correct location of ca-c…
Merge pull request #3027 from datailor-fr/f/3010-fix-ssl-issue
fix(#3010) Add symbolic link to the correct location of ca-cert file
fix(#3010) Add symbolic link to the correct location of ca-cert file
ymazzer commentedon Nov 27, 2024
@sachilles this issue is fixed, you can close it.
sachilles commentedon Nov 28, 2024
@ymazzer Thanks for your reminder!
Revert "fix(sameersbn#3010) Add symbolic link to the correct location…
Remove symbolic link to the correct location of ca-cert file
Remove symbolic link to the correct location of ca-cert file
Remove symbolic link to the correct location of ca-cert file
fix(sameersbn#3010) Add symbolic link to the correct location of ca-c…
Remove symbolic link to the correct location of ca-cert file