Skip to content

Commit

Permalink
update AWS docs
Browse files Browse the repository at this point in the history
  • Loading branch information
thisisaaronland committed Sep 9, 2022
1 parent 14ed150 commit 4105444
Showing 1 changed file with 91 additions and 7 deletions.
98 changes: 91 additions & 7 deletions AWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ See `Populate` section below.
The easiest way to populate the findingaid is to use the `create-dynamodb-import` tool to create a CSV file of all the findingaid pointers which can be used to create (and populate) a new "findingaid" table in DynamoDB.

```
$> /bin/create-dynamodb-import /usr/local/data/whosonfirst-findingaids/data/* > findingaid.csv
$> ./bin/create-dynamodb-import /usr/local/data/whosonfirst-findingaids/data/* /usr/local/data/whosonfirst-findingaids-venue/data/* > findingaid.csv
$> gzip findingaid.csv
$> aws --profile {PROFILE} s3 cp findingaid.csv.gz s3://{BUCKET}
```
Expand All @@ -18,21 +18,110 @@ Follow the instructions for importing the CSV file, specifying a new table calle

* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.Requesting.html#S3DataImport.Requesting.Console

Importing 23,310,204 findingaid pointers in September, 2022 took about 75 minutes.

_To do: Add notes about populating a standalone instance of DynamoDB outside of AWS._

## Sync / update

```
$> cd /usr/local/whosonfirst/whosonfirst-findingaids
$> make docker
```

### ECS

_ECS documentation is incomplete._

#### Tasks

```
/usr/local/bin/update-findingaids.sh,-T,awsparamstore://whosonfirst-findingaid-github-token?region=us-east-1&credentials=iam:,-O,3600
```

### IAM

###
#### Policies

##### FindingaidDynamoECS

```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListAndDescribe",
"Effect": "Allow",
"Action": [
"dynamodb:List*",
"dynamodb:DescribeReservedCapacity*",
"dynamodb:DescribeLimits",
"dynamodb:DescribeTimeToLive"
],
"Resource": "*"
},
{
"Sid": "SpecificTable",
"Effect": "Allow",
"Action": [
"dynamodb:BatchGet*",
"dynamodb:DescribeStream",
"dynamodb:DescribeTable",
"dynamodb:Get*",
"dynamodb:Query",
"dynamodb:Scan",
"dynamodb:BatchWrite*",
"dynamodb:Update",
"dynamodb:PutItem"
],
"Resource": "arn:aws:dynamodb:*:*:table/findingaid"
}
]
}
```

##### WOFParameterStoreFindingAidGithub

```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ssm:DescribeParameters"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Sid": "",
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": "arn:aws:ssm:{REGION}:{ACCOUNT}:parameter/whosonfirst-findingaid-github-token"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": [
"arn:aws:kms:{REGION}:{ACCOUNT}:key/CMK"
]
}
]
}
```

#### Roles

##### WOFECSFindingAid

* FindingaidDynamoECS
* WOFParameterStoreFindingAidGithub
* AmazonECSTaskExecutionRolePolicy

### EventBridge

## Resolver(d)

### Lambda
Expand Down Expand Up @@ -96,8 +185,3 @@ Create a new "REST" API and configure it with a new `{proxy+}` resources.
Delete the `ANY` method and then create a new `GET` method and configure it to point to the `FindingaidResolverServer` Lambda function.

Deploy the new API with a new stage name called "findingaid". When testing you should see something like this (assuming the findingaid hasn't been populated yet:

```
$> curl -s https://{PREFIX}.execute-api.{REGION}.amazonaws.com/findingaid/id/123456
Failed to get record for 123456, item {map[id:123456 repo_name:] map[id:123456 repo_name:] {<nil> <nil> 0} []} not found (code=NotFound)
```

0 comments on commit 4105444

Please sign in to comment.