Backup to S3 without proper credentials hangs

[gkocur]

I was trying to perform backup to S3 bucket, but I didn't provide AWS credentials
dcos cassandra backup start --external_location=s3:somebucket —backup_name=backup01

The backup status stuck in IN_PROGRESS state:

{
"id": "075e0950-a869-4fd6-8138-17521e32b330",
"name": "Snapshot",
"steps": [
{
"id": "dc290c90-d7a6-4576-a5bf-ab3dc4f1c4fb",
"status": "COMPLETE",
"name": "snapshot-node-0",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-0 [dc290c90-d7a6-4576-a5bf-ab3dc4f1c4fb]' has status: 'COMPLETE'."
},
{
"id": "8bfc2ab2-7ccd-40c3-a0ea-b9d9f84a01fd",
"status": "COMPLETE",
"name": "snapshot-node-1",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-1 [8bfc2ab2-7ccd-40c3-a0ea-b9d9f84a01fd]' has status: 'COMPLETE'."
},
{
"id": "34437cac-989d-4efd-a739-6611aba22d19",
"status": "COMPLETE",
"name": "snapshot-node-2",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-2 [34437cac-989d-4efd-a739-6611aba22d19]' has status: 'COMPLETE'."
}
],
"status": "COMPLETE"
},
{
"id": "0dc5bda7-4d81-4692-ac34-da694183c878",
"name": "Upload",
"steps": [
{
"id": "e1f7f292-7bda-4010-a49f-2541d5eea729",
"status": "IN_PROGRESS",
"name": "upload-node-0",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-0 [e1f7f292-7bda-4010-a49f-2541d5eea729]' has status: 'IN_PROGRESS'."
},
{
"id": "175aa0b7-79f5-46fb-b10e-0c9381238c0f",
"status": "PENDING",
"name": "upload-node-1",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-1 [175aa0b7-79f5-46fb-b10e-0c9381238c0f]' has status: 'PENDING'."
},
{
"id": "b3dbefbc-0b74-43b2-83a0-685871c3d0a6",
"status": "PENDING",
"name": "upload-node-2",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-2 [b3dbefbc-0b74-43b2-83a0-685871c3d0a6]' has status: 'PENDING'."
}
],
"status": "IN_PROGRESS"
},
{
"id": "6900fde9-05c8-42ad-9639-c5564a0b5268",
"name": "BackupSchema",
"steps": [
{
"id": "75190241-8eb0-401d-a1d1-0647a2b43ac9",
"status": "PENDING",
"name": "backupschema-node-0",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-0 [75190241-8eb0-401d-a1d1-0647a2b43ac9]' has status: 'PENDING'."
},
{
"id": "7e080380-a310-4ac9-9fb8-bbc1c639ed50",
"status": "PENDING",
"name": "backupschema-node-1",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-1 [7e080380-a310-4ac9-9fb8-bbc1c639ed50]' has status: 'PENDING'."
},
{
"id": "70e98253-22b6-4538-aae5-8327752c709f",
"status": "PENDING",
"name": "backupschema-node-2",
"message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-2 [70e98253-22b6-4538-aae5-8327752c709f]' has status: 'PENDING'."
}
],
"status": "PENDING"
}
],
"errors": [],
"status": "IN_PROGRESS"
}

I stopped this backup with dcos cassandra backup stop.
After stopping it dcos cassandra backup status shows no running/pending backup but upload-node-0 task is still running.

Logs from upload-node-0 task:

ERROR [2017-03-09 11:19:30,488] com.mesosphere.dcos.cassandra.executor.tasks.UploadSnapshot: Upload snapshot failed
! java.lang.NullPointerException: null
! at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.getPrefixKey(S3StorageDriver.java:62) ~[cassandra-executor.jar:na]
! at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.upload(S3StorageDriver.java:142) ~[cassandra-executor.jar:na]
! at com.mesosphere.dcos.cassandra.executor.tasks.UploadSnapshot.run(UploadSnapshot.java:82) ~[cassandra-executor.jar:na]
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

[triclambert]

This repo is deprecated and will be archived in one week. Please see the latest version of Cassandra or DSE for DC/OS:

https://docs.mesosphere.com/service-docs/cassandra/
https://docs.mesosphere.com/service-docs/dse/ (enterprise-only)