AWS Infrastructure

[esheehan-gsl]

@ian-noaa I’m planning to build the UI as a single-page application (SPA), so the hosting is relatively simple. In production, it will just be some HTML, CSS, and JavaScript files that will need to be served over HTTP. I think for AWS this means we’ll need:

• S3 for storing the files
• CloudFront for hosting the files
• API Gateway so that the JavaScript running the browser can fetch new data from whatever backend services we create

Alternatively, we could go for Amplify which is — I believe — a service that Amazon offers to wire together all of the things you need for hosting an app like this and automating your deploys.

For CI, I’ll just need an environment with NodeJS to build the application into static files and copy them to S3. (Amplify may have tools for CI as part of the package, I’m unclear on that point.) This is actually going to look an awful lot like the process for building the SOS website, so you can probably look at our GitLab CI files to see how the build process works.

Let me know if you need any more information, or if you have any concerns about how we’re going to connect the UI to the backend.

[esheehan-gsl]

It occurred to me that you were considering a Kubernetes cluster for handling data. We could also host the UI out of an nginx or Apache container in a cluster as well, if that ends up making more sense. I’m not sure that we’ll see any significant trade-offs in performance or reliability with that approach compared to a CDN, so this is an option we can consider.

[ian-noaa]

I was looking at Kubernetes since we were looking at using Jebb's GRIB to Zarr converter and that's where that runs. The other reason I was considering Kubernetes is it would theoretically make it easier to run on prem if needed as we'd have containers we could use there. I do wonder if we'd be able to hook a CDN up to the website in the Kubernetes cluster if it was needed.

[esheehan-gsl]

I think if we used a CDN for static assets—like the HTML, CSS, and JS for the UI—that stuff wouldn’t be in the cluster. In that situation, we’d need something (API Gateway?) that exposed URLs over HTTP that we could fetch() data from in the UI. If the UI is being hosted from a container in the cluster, then it can probably make those HTTP requests without exposing the API outside of the cluster and all we’d need is a way to resolve URLs to the application so that they are handled by the web server running in the cluster (which may still require API Gateway?).

I think we are going to need to be running Jebb’s converter, so we will probably need this. Do you know if it’s possible to spin up containers in response to an S3 event? I’m pretty sure S3 events can trigger Lambda functions. It’d be great if we can avoid polling for new data.

[ian-noaa]

Gotcha. If we had the CDN for static assets outside of the cluster I believe we'd use a Kubernetes ingress controller to expose the services in the cluster to the CDN frontend. But there are a good number of ways to do this so it's probably a good thing to discuss with the VLab folks to see what they're familiar with and would recommend. You're correct that if the UI is a container in the cluster, then we can manage this all through Kubernetes. They've got an example of one way to do this in the Kubernetes documentation.

It's definitely possible to spin up containers in response to an S3 event. You set up an SQS queue that S3 adds events to and that the cluster can react to. Jebb's converter does this to add containers and scale the cluster up & down as files come in.

[ian-noaa]

Sounds good! I think that gives me what I need to get started.

Regarding using API Gateway - I recall you mentioning you were interested in a GraphQL API for the backend. Was that correct? It sounds like API Gateway only supports REST or Websockets. However, AWS has another product called AppSync that we could look at.

I hadn't heard about Amplify before but would also be happy to look into it if you wanted to go that way.

Regarding CI - that sounds good. I was originally thinking we would use GitHub Actions for CI. How does that sound to you?

[esheehan-gsl]

I was thinking that GraphQL might be useful because it lets the client define the contents of the response when it requests data, so it might be easier to build out a more flexible API and minimize payload sizes. But I’m not sure at this point if that will even be all that important to us. Even if we did decide to go that route, it should be something we can just add on to our backend if we want it; there are Python libraries we can use to help us build a GraphQL endpoint in Flask.

So I’m inclined to skip AppSync for now.

---

GitHub Actions are fine with me, although I’m unclear on how to connect them to AWS. I think that AWS has its own CI utilities that it wants you to use for building and deploying by giving it access to your GitHub repo. At least, that seems like an option for CI with Amplify. I remember being a little perplexed by this when I was trying to get the HRRR Smoke application going on AWS.

I’m fine either way, as long as we run tests and builds automatically when we merge to main.

[ian-noaa]

Makes sense regarding GraphQL/AppSync!

For GitHub Actions - we basically give GitHub (as a CI secret) the AWS access & secret keys for a deploy account so a GitHub runner can then go through and update resources. The other method would be to do CI through GitHub Actions to create images and stick them in a registry. We'd then use something like Flux or Argo to do the actual deployment into the cluster.

AWS does have code hosting and build/deploy tools but I'm less familiar with them.

[esheehan-gsl]

@ian-noaa do you have an estimate of when we’ll have some infrastructure available? I’m hoping to scaffold the frontend next week and it would be nice to work on the deploy as well.

[ian-noaa]

Not yet. Unfortunately, the Hackathon has been consuming most of my time. I'll work on setting up a meeting with the VLab team this week to let them know what we're currently thinking for infrastructure as well as to get their input on solutions.

The Hackathon (hopefully) launches next week so I'll have more time to put into this after that.

[terraladwig]

Please add me to the meeting with the VLab folks (Ken Sperow) about the AWS resources. So far we have talked with them about the cluster design for the RTMA runs on the cloud. But the infrastructure you are requesting is probably quite different.

[ian-noaa]

Sounds good! I'll ask them to add you. And I agree - my guess is the infrastructure we need will be pretty different from the needs for the HPC cluster.

[esheehan-gsl]

I created an issue (#12) for getting our infrastructure set up so that we can track that work. We can adjust the tasks as neeeded.