Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First implementation #2

Merged
merged 3 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ on:
push:
branches: main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
S3_BUCKET: aws-s3-lock-ci

jobs:
build:
runs-on: ubuntu-latest
Expand All @@ -17,3 +24,53 @@ jobs:
- run: yarn install
- run: yarn build
- run: yarn test --passWithNoTests

integration:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ secrets.AWS_ROLE }}

- id: lock-1
name: Acquire a lock
uses: ./
with:
bucket: ${{ env.S3_BUCKET }}
expires: 5s

- id: lock-2
name: Wait on previous lock then acquire our own
uses: ./
with:
bucket: ${{ env.S3_BUCKET }}
timeout: 10s

- name: Verify locks
run: |
[[ -n '${{ steps.lock-1.outputs.acquired-at }}' ]] # lock-1
[[ -n '${{ steps.lock-2.outputs.acquired-at }}' ]] # lock-2
# lock-2 acquired at should be +5s

integration-post:
needs: integration
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ secrets.AWS_ROLE }}
- name: Assert lock was released
uses: ./
with:
bucket: ${{ env.S3_BUCKET }}
timeout: 0s
13 changes: 0 additions & 13 deletions .github/workflows/example.yaml

This file was deleted.

2 changes: 1 addition & 1 deletion .restyled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ restylers:
- whitespace:
include:
- "**/*"
- "!dist/index.js"
- "!dist/**/*"
- "*"
71 changes: 61 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,75 @@
# TypeScript Action Template
# AWS S3 Lock

Our custom template repository for GitHub Actions implemented in TypeScript.

[Creating a repository from a template][docs].

[docs]: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template

**NOTE**: Be sure to look for strings like "TODO" or "Action name" and update
them accordingly.
Wait for, acquire, and release a distributed lock via S3 storage.

## Usage

```yaml
- uses: freckle/TODO-action@v1
- uses: aws-actions/configure-aws-credentials@v4
with:
# ...

# This blocks until the lock is acquired, or errors if timeout is reached
- uses: freckle/aws-s3-lock-action@v1
with:
# Required
bucket: an-existing-s3-bucket

# Optional, defaults shown
# name: {workflow}/{job}
# expires: 15m
# timeout: {matches expires}
# timeout-poll: 5s

- run: echo "Lock held, do work here"
```

![](./screenshot.png)

The lock is released (the S3 object deleted) in our Post step, which provides a
pretty robust guarantee of release. Expired locks are ignored (not deleted), so
it's recommended you put a Lifecyle policy on the Bucket to clean them up after
some time.

## Inputs and Outputs

See [action.yml](./action.yml) for a complete list of inputs and outputs.

## Implementation Details

### Algorithm

This tool implements a version of the locking algorithm described in this
[StackOverflow answer][answer].

[answer]: https://stackoverflow.com/questions/45222819/can-pseudo-lock-objects-be-used-in-the-amazon-s3-api/75347123#75347123

- Upload a lock object to S3 at `<name>.<created>.<uuid>.<expires>`

All time values are milliseconds since epoch.

- List all other lock objects (prefix `<name>.`)

Filter out any expired keys (looking at `expires`) and sort, which implicitly
means by `created` then `uuid` as desired.

- If the first one (i.e. oldest) is our own, we've acquired the lock
- If not, we lost the race; remove our object, wait, and try again

### Ordering Semantics

Each time we attempt to acquire the lock, we create a new key name (e.g.
`<created>` and `<expires>` both change), this effectively loses our "place in
line" but ensures that expiry is measured from time of acquisition and not time
of first attempt. There are trade-offs either way, and possible room for
improvement, so this is just how we're doing it for now.

### Caveat

**This tool is not meant to be bullet-proof**. We built it for our needs and
accept that there are simply no strong guarantees in this locking mechanism's
operation at scale. Your mileage may vary; patches welcome.

## Versioning

Versioned tags will exist, such as `v1.0.0` and `v2.1.1`. Branches will exist
Expand Down
39 changes: 33 additions & 6 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,38 @@
name: "Action name"
description: "Action description"
name: aws-s3-lock
description: Wait on, acquire, and clean-up S3-based locks
author: "Freckle"
inputs:
github-token:
description: "Override GitHub token, if necessary"
bucket:
description: |
Name of an existing S3 bucket to use.
required: true
default: "${{ github.token }}"
name:
description: |
Name for the lock object. Include any prefix you want within the bucket.
The key will be built as "name.created.uuid.expires".
required: true
default: "${{ github.workflow }}/${{ github.job }}"
expires:
description: |
How long before concurrent operations consider this lock expired.
required: true
default: 15m
timeout:
description: |
How long to wait for the lock to become available. Default matches
expires.
required: false
timeout-poll:
description: |
How long to wait between attempts for the lock. Default is 5s.
required: true
default: 5s
outputs:
key:
description: "Key of the S3 object representing the lock"
acquired-at:
description: "Timestamp the lock was acquired"
runs:
using: "node20"
main: "dist/index.js"
main: "dist/acquire/index.js"
post: "dist/release/index.js"
33 changes: 33 additions & 0 deletions dist/acquire/index.js

Large diffs are not rendered by default.

Loading