Cannot list more than 100 directories or 1000 files with S3 Protocol on Google Cloud Storage bucket

[Mastergalen]

Describe the bug

When connecting Cyberduck to a Google Cloud Storage bucket using interoperability mode with the S3 protocol, Cyberduck only lists up to 100 folders and up to 1000 files. The rest of them are missing.

Why am I not just using the Google Cloud Storage connection method?

Because Cyberduck's OAuth scopes are too broad, requiring full devstorage.full_control permissions, I am asked by our Security team to limit access by using HMAC keys tied to service accounts with minimal permissions.

Expected behavior

All files and folders should be listed.

Desktop (please complete the following information):

• OS: macOS Sonoma 14.7
• Version 9.0.2 (42108)

Log Files
Please include the log file. You can reach the logging output in Preferences → Connection.

No log files

Additional context
Add any other context about the problem here.

[dkocher]

As another workaround you may want to create a custom connection profile with limited scopes. See Custom OAuth 2.0 Client ID for Google Cloud Storage and Google Drive for a template.

[Mastergalen]

Thanks for pointing out this feature. Sadly only one OAuth client can be created per Google Project, and we are already using the OAuth consent screen for another app in the same project where all our buckets reside in.

Furthermore, the ability to tie HMAC keys to specific GCP service accounts is an easier way of authenticating users to buckets.

[dkocher]

We do not currently support paginating using ListBuckets when listing buckets, however the number of keys retrieved should not be limited and is using chunking with a size of 1000.

[Mastergalen]

Thanks @dkocher for replying.

So ListBuckets is for listing all the buckets owned by the authenticated user. That's fine as my account doesn't have that many buckets to list and all of them are being successfully listed.

It's the listing of objects inside a bucket that seems to be limited to only 1000 where the pagination is not working properly.

[dkocher]

Regarding the devstorage.full_control scope requested ¹. You may change this to devstorage.read_write instead in a custom connection profile. By default we require devstorage.full_control to support configuring ACLs in Cyberduck.

Footnotes

1. https://developers.google.com/identity/protocols/oauth2/scopes ↩