Proposal for Tool/Workflow Secret Management Implementation

[davelopez]

After an initial discussion with @bgruening and @arash77, this proposal expands on #1751 to facilitate a team discussion on design and implementation strategies. An initial prototype is underway at #19084.

Overview

The goal is to enable Galaxy tools to securely declare, request, and manage secret tokens or credentials that users can provide. This proposal outlines a way for tool authors to define credential requirements, a user interface (UI) to request and manage credentials, and the backend to store and retrieve them securely.

Declaring Tool Secrets

Some options for declaring tool secrets in the tool’s XML are:

Option A: Within the <requirements> section

Tool authors will be able to declare that a tool requires credentials by adding a <secret> element within the <requirements> section, for example:

<requirements>
    <secret type="token" reference="some/unique/reference" inject_as_env="target_env_var" label="API Key" description="This is the API Key for blah…" required="true"/>
</requirements>

The fields are defined as follows:

• type: Supports “token” for simple API keys and potentially “credentials” for username+password pairs.
• reference: A unique path-like string for the Vault key. This will default to a prefixed form of the tool ID and version to avoid accidental sharing but can be overridden for cases where secrets should be shared across tools.
• inject_as_env: Specifies the environment variable where the credentials should be injected.
• label and description: Provide information to display in the UI when requesting credentials from users.
• required: A boolean that indicates if the credential is mandatory or if the tool can run with anonymous access.

Option B: Within a dedicated <secrets> section supporting multiple service credentials

We could instead introduce a dedicated <secrets> section, making it easier to distinguish credentials from other tool requirements.

For tools that require multiple credentials or credentials with multiple parts (e.g., username and password):

<secrets>
  <credentials name="service1" reference="some/unique/reference" required="true" description="Please provide credentials for blah…">
    <token inject_as_env="target_env_var_apikey" label="API Key" description="This is the API Key for blah…" />
  </credentials>
  <credentials name="service2" reference="some/other/unique/reference" required="false">
    <token name="username" inject_as_env="target_env_var_user" label="Your username" />
    <token name="password" inject_as_env="target_env_var_pass" label="Your password" />
  </credentials>
</secrets>

Requesting Tool Secrets in the UI

In the Tool Form

When a tool that requires credentials is opened, a warning banner will appear at the top of the Tool Form. This banner will display a button that, when clicked, opens a dialog where users can input the required credentials. If credentials have already been provided, the banner will turn green with a message like “You have already provided credentials for this tool,” along with an option to edit them.

In the Workflow Run Form

When a workflow containing a tool that requires credentials is run, the user will be prompted to provide the necessary credentials before the workflow can be executed. This will be similar to the tool form, with a warning banner at the top of the form and a dialog to input credentials. When requesting credentials for a workflow, the system will check if the user has already provided credentials for all tools in the workflow (and sub-workflows) and only prompt for missing credentials.

Backend: Storing and Managing Tool Secrets

Once provided by the user, credentials will be stored in the user Vault, with references saved in a new table in the Galaxy database called user_tool_secrets. This table will contain at the very minimum:

• User ID
• Tool ID
• Tool Version
• Vault reference (unique "path") for the credential

This enables the system to check whether the user has provided credentials for a tool and to update the UI accordingly.

Users will manage their credentials in a new section in their user preferences. This section will list all tools for which the user has provided credentials and allow them to edit, delete, or copy credentials between tools if needed.

API Endpoints for Tool Secrets Management

To support these operations, we propose adding the following API endpoints for discussion:

• GET /api/user/{id}/tool_secrets - Lists all credentials the user has provided (credentials themselves are not included).
• GET /api/user/{id}/tool_secrets/{tool_id}/{tool_version} - Verifies if credentials have been provided for a specific tool (no credential data returned).
• POST /api/user/{id}/tool_secrets/{tool_id}/{tool_version} - Allows users to provide credentials for a tool or copy them from another tool or version.
• PUT /api/user/{id}/tool_secrets/{tool_id}/{tool_version} - Updates credentials for a specific tool.
• DELETE /api/user/{id}/tool_secrets/{tool_id}/{tool_version} - Deletes credentials for a specific tool.

Open Questions

• Q1*: Placement of <secret> Element. Should we introduce a dedicated section in the tool’s XML instead of including secrets in the section? This might make the code more readable and organized.

• Q2*: Vault as a Requirement. Since all secrets should be securely stored, we assume the Vault is required. Should we make this a hard requirement or allow configurations that skip the Vault, storing secrets in user preferences instead? This flexibility might benefit testing or smaller Galaxy instances that don’t need a Vault, though it will make it not production-ready.

Other alternatives?

Please share your thoughts on the proposed design and suggest any alternatives or improvements.

[mvdbeek]

Q1*: Placement of Element

I would keep it in requirements, but I don't think this is a critical decision

• Q2*: Vault as a Requirement. Since all secrets should be securely stored, we assume the Vault is required. Should we make this a hard requirement or allow configurations that skip the Vault, storing secrets in user preferences instead? This flexibility might benefit testing or smaller Galaxy instances that don’t need a Vault, though it will make it not production-ready.

In a dev environment I would probably enabled the database-backed vault ?

Generally speaking I would prefer to not call it tool secrets, and I wouldn't limit it to tools. You could think of workflow actions that are not tool steps that require secrets, like when you'd interact with an external service e.g. for exports after a workflow has completed. I would maybe make a user_secrets table with uuid as the primary key plus the corresponding CRUD endpoints. You can still filter by tool / tool version in the endpoint, but I wouldn't make it part of the path operation.

[davelopez]

Thank you Marius for the feedback!

So, for the XML elements, what do you think about this?:

<requirements>
  <credentials name="service1" reference="some/unique/reference" required="true" label="My awesome API" description="Please provide credentials for blah…">
    <secret name="my_token" inject_as_env="target_env_var_apikey" label="API Key" description="This is the API Key for blah…" />
  </credentials>
  <credentials name="service2" reference="some/other/unique/reference" required="false">
    <secret name="s2_username" inject_as_env="target_env_var_user" label="Your username" />
    <secret name="s2_password" inject_as_env="target_env_var_pass" label="Your password" />
  </credentials>
</requirements>

• The credentials element is a container that let you group secret elements.
  • The name attribute is optional and can be used to identify the credentials group or display a label in the UI.
  • The reference attribute is required and is used to store the credentials in the Vault. If you update the tool version but don't alter this reference, the tool will reuse the credentials the user provided for the previous version otherwise, altering this reference will force the user to provide new credentials.
  • The required attribute is optional and defaults to false if the service can work without credentials.
  • The label and description attributes are optional and provide information to display in the UI when requesting credentials from users.
  • The secret element is used to define the individual credentials.
    • The name attribute is required and is used to identify the credential. The vault key will be a combination of the service reference and secret name attributes (i.e. some/unique/reference/my_token).
    • The inject_as_env attribute is required and specifies the environment variable where the credentials should be injected.
      • ⚠️Question: How to make sure we don't overwrite existing environment variables?
    • The label and description attributes are optional and provide information to display in the UI when requesting credentials from users.

For the database schema, what about this?:

erDiagram
  USER_SECRETS {
    uuid id PK
    int user_id
    string service_reference UK
    string secret_name
  }

Loading

The USER_SECRETS table will store the Vault reference for each individual secret provided by the user.

• The user_id column is a foreign key to the galaxy_user table.
• The Vault reference is a combination of service_reference and secret_name columns (i.e. {service_reference}/{secret_name}).

This should be generic enough. There's probably no need to store the tool ID and version in the database, as we can get this information from the tool's XML requirements and then check if the user has provided credentials for that service/secret.

And finally the API endpoints could look like this:

• GET /api/user/{id}/secrets - Lists all credentials the user has provided (credentials themselves are not included).
• GET /api/user/{id}/secrets/{service_reference} - Verifies if credentials have been provided for a specific service (no credential data returned).
• GET /api/user/{id}/secrets/{service_reference}/{secret_name} - Verifies if credentials have been provided for a specific secret (no credential data returned).
• POST /api/user/{id}/secrets/{service_reference}/{secret_name} - Allows users to provide credentials for a secret.
• PUT /api/user/{id}/secrets/{service_reference}/{secret_name} - Updates credentials for a specific secret.
• DELETE /api/user/{id}/secrets/{service_reference} - Deletes all credentials for a specific service.
• DELETE /api/user/{id}/secrets/{service_reference}/{secret_name} - Deletes credentials for a specific secret.

[hexylena]

reference: A unique path-like string for the Vault key. This will default to a prefixed form of the tool ID and version to avoid accidental sharing but can be overridden for cases where secrets should be shared across tools.

I didn't see any examples of this, but I'd be interested to, since the use case would be relevant for @abretaud and myself. E.g. the suite of Apollo tools would all want to re-use the same credentials because they all talk to the same backend, even across multiple tools.

So I'd be looking at writing something like this, yes?

<requirements>
  <credentials name="Apollo" reference="gmod.org/apollo" required="true">
    <secret name="server" inject_as_env="apollo_url" label="Your Apollo server" />
    <secret name="username" inject_as_env="apollo_user" label="Your Apollo username" />
    <secret name="password" inject_as_env="apollo_pass" label="Your Apollo password" />
  </credentials>
</requirements>

some questions based on this use case:

• If i'm reading this correctly, this assume one set of credentials, permanently? What if they want to have multiple credentials for multiple servers? Do they need to change every value between uses?
• In this use case there's also several sensitivity levels, server (non-sensitive), user (somewhat), password (shouldn't be shown). Is there a plan for handling that? It would be nice for e.g. apollo url to be visible to the user so they can be sure it's the right one.

That UX I'm not sure if it makes sense for me, given how all of the tools I manage that require credentials, would be used. Apollo and the XNAT tool we're working on all have the built in assumption that they might want to talk to different servers at some point. So I think I expected this to be a tuple, sort of. That the user would be asked for credentials, they would provide a set of secrets for a specific credential, and then save that with a name. Then when the user needs to run the tool, they can select between tuples with their nicknames, to say "I want to use the dev apollo" or "the production apollo".

[mvdbeek]

I'm not sure I agree,

I agree with David's proposal. We've discussed this at the backend meeting and we're going to start out simple. Any sort of advanced sharing is out of scope initially.

re-using the credential name as another benign tool.

This is not possible using tool shed installed tools.

[hexylena]

We've discussed this at the backend meeting and we're going to start out simple. Any sort of advanced sharing is out of scope initially.

oh, then I misunderstood "facilitate a team discussion", my apologies! If the scope is set in stone already, please feel free to ignore my comments.

I genuinely understand needing to split up work into reasonable units. Just, if it's a unit of work that ends with "now tool authors should use this here is the documentation", just want to check it is something that's useful for me as a tool author?

Do you see a path to sharing the credentials across tools in a future version? Can that sort of implementation decision be pushed off with migrations? (I'm not primarily a dev, I wouldn't know.) Would the existing credentials will be shown to the user? Or would it not find their credentials and they would have to re-enter them?

This is not possible using tool shed installed tools.

yes, but I have that. Currently. This would be a regression for me to switch to.

I fear that this "advanced sharing" isn't an advanced use case, rather it is the most common one. The apollo and tripal tools work like that now, the user provides credentials once and they're re-used. The IUC Beacon2 suite does as well, OMERO as well. EGA is an outlier, but we've considered adding more tools to that suite for better user experience.

[mvdbeek]

When a user complains it's too much work to pass credentials to a tool we can consider the next step, but I wouldn't invert this order with over-engineering a complex solution. Surely there can be a front-end hint asking if a similar credential type can be used, I don't think we need to plan this form the start.

[hexylena]

If that's the decision, I understand. We can end the discussion. I just wanted to consider the requirements of those who are heavy tools-with-credentials users like me. I will probably stick to user preferences then, for my suites, and perhaps I can use this if I ever write individual tools.

[mvdbeek]

It's not like there is a decision and we will not alter any of it, we only discussed this aspect at the backend meeting with the outcome that we will focus on a simple and safe implementation of a feature we currently do not have.

Surely there can be a front-end hint asking if a similar credential type can be used

Would you agree that this is a workable compromise between being explicit and convenient ? The feature itself will be considerably more discoverable both by admins and users, so I hope that you'll consider it for your tool suites as well. Given there isn't much of an annotation you could of course continue to have the current mechanism as a fallback.

[bernt-matthias]

My only bit to add would be that it might be cool to have optional credentials, e.g. for services that allow anonymous login.

[davelopez]

We will try to support required="false" for this use case as in the examples above.

[bernt-matthias]

Cool. Would argue for optional="true" for consistency (e.g. param).

[nuwang]

Also wanted to add this for ref: #15300
The OIDC token is injected via the <environment_variables> section there. Should these have some unified mechanism?