add support for specifying bucket name as mapping with repo id as key

[boegel]

With this in place, we should be able to specify the bucket in the bot configuration as follows:

bucket_name = {
    "eessi-hpc.org-2023.06-compat": "eessi-staging-2023.06",
    "eessi-hpc.org-2023.06-software": "eessi-staging-2023.06",
    "eessi.io-2023.06-compat": "software.eessi.io-2023.06",
    "eessi.io-2023.06-software": "software.eessi.io-2023.06",
}

which effectively allows us to let the bot use different buckets for different target repos (and even different buckets for different repo versions, or different buckets for compat vs software layer, etc.)

By checking the value type of bucket_name, we can ensure that current bot instances that use a hardcoded bucket name as string value will not be affected, so this change is backwards-compatible.

[boegel]

Marked this a draft, testing showed it doesn't work yet, seeing this somehow:

[20231125-T15:58:14] Using specified bucket: {
"eessi-hpc.org-2023.06-compat": "eessi-staging-2023.06",
"eessi-hpc.org-2023.06-software": "eessi-staging-2023.06",
"eessi.io-2023.06-compat": "software.eessi.io-2023.06",
"eessi.io-2023.06-software": "kh-eessi-staging" }

[20231125-T15:58:14] run_subprocess(): 'Upload tarball to S3 bucket' by running '/home/boegel/eessi-bot-software-layer/scripts/eessi-upload-to-staging --bucket-name {
"eessi-hpc.org-2023.06-compat": "eessi-staging-2023.06",
"eessi-hpc.org-2023.06-software": "eessi-staging-2023.06",
"eessi.io-2023.06-compat": "software.eessi.io-2023.06",
"eessi.io-2023.06-software": "kh-eessi-staging" } --endpoint-url URL_TO_S3_SERVER --repository boegel/software-layer --pull-request 29 /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1214/eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz' in directory '/home/boegel/eessi-bot-software-layer'

[20231125-T15:58:14] run_cmd(): Error running '/home/boegel/eessi-bot-software-layer/scripts/eessi-upload-to-staging --bucket-name {
"eessi-hpc.org-2023.06-compat": "eessi-staging-2023.06",
"eessi-hpc.org-2023.06-software": "eessi-staging-2023.06",
"eessi.io-2023.06-compat": "software.eessi.io-2023.06",
"eessi.io-2023.06-software": "kh-eessi-staging" } --endpoint-url URL_TO_S3_SERVER --repository boegel/software-layer --pull-request 29 /home/boegel/eessi-bot-software-layer/jobs/2023.11/pr_29/1214/eessi-2023.06-software-linux-x86_64-generic-1700927661.tar.gz' in 'None
           stdout ''' is not a readable non zero-sized file.
'
           stderr '/bin/sh: line 1: eessi-hpc.org-2023.06-compat:: command not found
/bin/sh: line 2: eessi-hpc.org-2023.06-software:: command not found
/bin/sh: line 3: eessi.io-2023.06-compat:: command not found
/bin/sh: line 4: eessi.io-2023.06-software:: command not found
'
           exit code 127

[boegel]

Problem fixed, successfully tested with boegel/software-layer#29

[trz42]

Changes look ok. A few suggestions (perfectly fine if we open an issue or some for them so we don't loose to much time implementing them):

• maybe rename bucket_name to bucket_spec?
• nice to have updates to PR comments if an error happens, but we should then have corresponding settings in app.cfg

I would either not do code improvements as in tasks/build.py or mention them in the PR description. Of course, generally code improvements are great.

[trz42]

OK to merge. Suggestions can be taken care of later.

[boegel]

@trz42 Can you open an issue for your suggestions?

Although I would be OK with renaming bucket_name to bucket_spec in the configuration file, that's a painful transition, and it may not be worth it. Even if bucket_name is a mapping of repo ID to bucket name, it still specifies a bucket name, just indirectly. So I'm fine with keeping bucket_name