Deleting both a storage bucket and a storage notification results in a 412 Error

[klnusbaum]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.6
on linux_amd64

• provider registry.terraform.io/hashicorp/google v6.11.1

Affected Resource(s)

google_storage_bucket
google_storage_bucket_iam_policy
google_storage_notification

Terraform Configuration

Before

locals {
  region = "us-central1"
}

resource "google_service_account" "accessor" {
  account_id = "pub-sub-test-service-account"
}

resource "google_storage_bucket" "test_bucket" {
  name          = "xxx-pubsub-test-bucket"
  location      = local.region
  storage_class = "STANDARD"

  uniform_bucket_level_access = true
  public_access_prevention    = "enforced"

  force_destroy = true
}

data "google_iam_policy" "bucket_policy_data" {
  binding {
    role = "roles/storage.admin"

    members = ["serviceAccount:${google_service_account.accessor.email}"]
  }
}

resource "google_storage_bucket_iam_policy" "bucket_policy" {
  bucket      = google_storage_bucket.test_bucket.name
  policy_data = data.google_iam_policy.bucket_policy_data.policy_data
}

resource "google_pubsub_topic" "topic" {
  name = "xxx-pubsub-test-bucket-topic"
}

resource "google_storage_notification" "storage_notification" {
  bucket         = google_storage_bucket.test_bucket.name
  payload_format = "JSON_API_V1"
  topic          = google_pubsub_topic.topic.id

  depends_on = [google_pubsub_topic_iam_policy.topic_policy]
}

data "google_storage_project_service_account" "gcs_account" {}

data "google_iam_policy" "topic_policy_data" {
  binding {
    role = "roles/pubsub.publisher"
    members = [
      "serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"
    ]
  }
}

resource "google_pubsub_topic_iam_policy" "topic_policy" {
  topic       = google_pubsub_topic.topic.name
  policy_data = data.google_iam_policy.topic_policy_data.policy_data
}

terraform {
  backend "gcs" {
    bucket = "chronosphere-audited-tf-state"
    prefix = "gcp/environments/matt-test-20240913-d74c"
  }
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "= 6.11.1"
    }
  }
  required_version = "1.9.6"
}

provider "google" {
  project                     = "<your project name>"
}

After:

terraform {
  backend "gcs" {
    bucket = "chronosphere-audited-tf-state"
    prefix = "gcp/environments/matt-test-20240913-d74c"
  }
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "= 6.11.1"
    }
  }
  required_version = "1.9.6"
}

provider "google" {
  project                     = "<your project name>"
}

Debug Output

If necessary, please reach out to our enterprise support account manager who we can provide with a private, redacted debug log.

Expected Behavior

The terraform change should apply cleanly.

Actual Behavior

The following error is output:

╷
│ Error: Error setting IAM policy for storage bucket "b/xxx-pubsub-test-bucket": googleapi: Error 412: At least one of the pre-conditions you specified did not hold., conditionNotMet
│ 
│ 
╵

Steps to reproduce

1. Create a project in the cloud console.
2. In the supplied terraform, replace <your project name> with the name of the project.
3. terraform apply the before terraform.
4. terraform apply the after terraform

Important Factoids

We did some debugging of our own and found out the following pieces of information:

1. When deleting a google_storage_notification resource, behind the scenes this triggers a change to the google_storage_bucket_iam_policy, and results in a change to the etag for the policy.
2. When planning, the gcp provider appears to record the current etag for the google_storage_bucket_iam_policy. It will only ever use this value when doing operations on the google_storage_bucket_iam_policy.

Given points 1 and 2, the following appears to be happening. When terraform runs, it first deletes the google_storage_notification resource, triggering a change to the google_storage_bucket_iam_policy resource and resulting in its etag changing. Then, terraform attempts to delete the google_storage_bucket_iam_policy, but in the delete request it specifies an etag precondition using the now-out-of-date etag it got when it first planned the change. This causes a 412 error.

We've reached out to GCP support and they say the above etag behavior is working as expected.

References

No response

b/388547047

[ggtisc]

Confirmed issue!

Resources can be created but not deleted due to the shared error on the 1st try. After a 2nd try resources were successfully deleted