Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README to align with v1 + Changing Output of Format and Label Action #4

Merged
merged 4 commits into from
Jan 8, 2024
Merged

Update README to align with v1 + Changing Output of Format and Label Action #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b0253b7
(docs): Add description
Jan 7, 2024
7b0dcde
(docs): Add architecture section
Jan 7, 2024
0f423d6
(docs): Expand on cloud-init explanation
Jan 8, 2024
1d4e0b9
(docs): Grammatical corrections
Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
253 changes: 145 additions & 108 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,63 @@
# EBS Bootstrap
# ebs-bootstrap

![Screenshot of Configuration File and Output](assets/ebs-bootstrap.png)

`ebs-bootstrap` is a tool that provides a **safe** and **as-code** approach for managing block devices on AWS EC2. It supports the following block device operations...

* **Format** a file system
* **Label** a file system
* **Resize** a file system
* **Mount** a block device
* Manage **ownership** and **permissions** of the mount point

Currently, the following file systems are supported for querying and modification...

* `ext4`
* `xfs`

Block device mappings can be [unpredictable](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html#device-name-limits) for AWS Nitro EC2 Instance types. `ebs-bootstrap` is equipped with the tools to recover the originally assigned block device mappings (`/dev/sd[a-z]`) from the dynamically allocated device names (`/dev/nvme[0-26]n1`) produced by **EBS** and **Instance Store** volumes.

## Build

`ebs-bootstrap` can be built locally regardless of the architecture of the host machine. This is facilitated by a multi-architecture Docker build process. The currently supported architechtures are `linux/amd64` and `linux/arm64`.
`ebs-bootstrap` is a **statically-compiled** binary that can be built for both `linux/amd64` and `linux/arm64`. This process is facilitated by a multi-architecture Docker build process.

```bash
# Specific Architecture
./build/docker.sh --architecture arm64
ls -la
... ebs-bootstrap-linux-aarch64
[~] ./build/docker.sh --architecture arm64
[~] ls -la
ebs-bootstrap-linux-aarch64

# All Architectures
./build/docker.sh
ls -la
... ebs-bootstrap-linux-aarch64
... ebs-bootstrap-linux-x86_64
[~] ./build/docker.sh
[~] ls -la
ebs-bootstrap-linux-aarch64
ebs-bootstrap-linux-x86_64
```

## Recommended Setup
## Installation

The latest binary of `ebs-bootstrap` can be downloaded from [GitHub Releases](https://github.com/reecetech/ebs-bootstrap/releases)

```
curl -L \
-o /tmp/ebs-bootstrap \
"https://github.com/reecetech/ebs-bootstrap/releases/latest/download/ebs-bootstrap-linux-$(uname -m)"
sudo install -m755 /tmp/ebs-bootstrap /usr/local/sbin/ebs-bootstrap
```

## Documentation

<a href="https://github.com/reecetech/ebs-bootstrap/wiki" target="_blank">
<img src="https://img.shields.io/badge/github_wiki-%2523121011.svg?style=for-the-badge&logo=github&logoColor=white&color=%23aa82e6">
</a>

## Use Cases

### `systemd`

The ideal way of operating `ebs-bootstrap` is through a `systemd` service. This is so we can configure it as a `oneshot` service type that executes after the file system is ready and after `clout-init.service` writes any config files to disk. The latter is essential as `ebs-bootstrap` consumes a config file that is located at `/etc/ebs-boostrap/config.yml` by default.
A potential way of operating `ebs-bootstrap` is through a `systemd` service. This is so we can configure it as a `oneshot` service type that executes after the file system is ready and `clout-init.service` writes any config files to disk. The latter is essential as `ebs-bootstrap` consumes a config file that is located at `/etc/ebs-boostrap/config.yml` by default.

`ExecStopPost=-...` con point torwards a script that is executed when the `ebs-bootstrap` service exits on either success or failure. This is a suitable place to include logic to notify a human operator that the configured devices failed their relevant healthchecks and the underlying application failed to launch in the process.
`ExecStopPost=-...` can point towards a script that is executed when the `ebs-bootstrap` service exits on either success or failure. This is a suitable place to include logic to notify an individual that the configured devices failed their relevant healthchecks and the underlying application failed to launch in the process.

```ini
[Unit]
Expand All @@ -34,7 +68,7 @@ After=local-fs.target cloud-init.service
Type=oneshot
RemainAfterExit=true
StandardInput=null
ExecStart=ebs-bootstrap
ExecStart=/usr/local/sbin/ebs-bootstrap
PrivateMounts=no
MountFlags=shared
ExecStopPost=-/etc/ebs-bootstrap/post-hook.sh
Expand All @@ -43,17 +77,9 @@ ExecStopPost=-/etc/ebs-bootstrap/post-hook.sh
WantedBy=multi-user.target
```

```
cat /etc/ebs-bootstrap/post-hook.sh
#!/bin/sh
if [ "${EXIT_STATUS}" = "0" ]; then
echo "🟢 Post Stop Hook: Success"
else
echo "🔴 Post Stop Hook: Failure"
fi
```
It is then possible to configure another `systemd` service to only start if the `ebs-bootstrap` service is successful. Certain databases support the ability to spread database chunks across multiple devices that need to be mounted to pre-defined directories with the correct ownership and permissions.

It is then possible to configure another `systemd` service to only start if the `ebs-bootstrap` service is successful. Certain databases support the ability to spread database chunks across multiple devices that need to be mounted to pre-defined directories with the correct ownership and permissions. In this particular use-case, the database could be configured as a `systemd` service that relies on the `ebs-bootstrap.service` to succeed before attempting to start. This can be achieved by specifiying `ebs-boostrap.service` as a dependency in the `Requires=` and `After=` parameters.
In this particular use-case, the database could be configured as a `systemd` service that relies on the `ebs-bootstrap.service` to succeed before attempting to start. This can be achieved by specifiying `ebs-boostrap.service` as a dependency in the `Requires=` and `After=` parameters.

```ini
[Unit]
Expand All @@ -75,101 +101,112 @@ WantedBy=multi-user.target

### `cloud-init`

`cloud-init` can be configured through EC2 User Data to write a config file to `/etc/ebs-boostrap/config.yml` through the `write_files` module.
By default, `ebs-bootstrap` consumes a configuration file located at `/etc/ebs-boostrap/config.yml`. `cloud-init` can be configured to write a config to this location, using the `write_files` module. Ensure that `ebs-bootstrap` is installed on your Instance via the process of baking it into your Golden AMI or downloading it early in the boot process, using the `bootcmd` module.

The NVMe Driver, for Nitro-based EC2 Instances, has an [established behaviour](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html) of dynamically renaming a block device, based on the order in which the device is attached to the EC2 instance. This order is unpredictable, thus it is recommended to label the volumes appropriately so that they can be referenced consistently in the `mounts` module. The `mounts` module is responsible for managing the `/etc/fstab` file, thus establishing a reliable mechanism for mounting external volumes at boot, independent of the block storage driver used by the EC2 instance.
The advent of Instance Store provided Nitro-enabled EC2 instances the ability to harness the power of high speed NVMe. For a stateful workload like a database, you might want a fast and ephemeral space for temporary tables, alongside a stateful EBS volume declared in a different CloudFormation Stack. However, these Instance Store devices were ephemeral and had to be formatted and mounted on each startup cycle.

From the perspective of a **sceptical** Platforms Engineer, you do not mind a tool like `ebs-bootstrap` automating the task of formatting and mounting an ephemeral device. However, you personally draw the line on automation executing modifications to a stateful device, **without** the prior consent of a human. `ebs-bootstrap` empowers this Platform Engineer by allowing them to specify the execution mode, on a **device-by-device** basis: Instance Store (`force`) and EBS Volume (`healthcheck`)

```yaml
Parameters:
LatestUbuntuAmi:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id

Resources:
Instance:
Type: AWS::EC2::Instance
...
Volumes:
- Device: /dev/sdb
VolumeId: !Ref ExternalVolumeID
UserData:
Fn::Base64: !Sub
- |+
#cloud-config
write_files:
- content: |
global:
mode: healthcheck
devices:
/dev/sdb:
fs: ${FileSystem}
mount_point: /mnt/app
owner: ec2-user
group: ec2-user
permissions: 755
label: external-vol
path: /etc/ebs-bootstrap/config.yml
mounts:
- [ "LABEL=external-vol", "/mnt/app", "${FileSystem}", "${MountOptions}", "0", "2" ]
- FileSystem: ext4
MountOptions: defaults,nofail,x-systemd.device-timeout=5
Properties:
...
ImageId: !Ref LatestUbuntuAmi
InstanceType: m5ad.large # Nitro Instance Type
Volumes:
- Device: /dev/sdb # EBS Volume
VolumeId: !ImportValue StatefulVolumeId
BlockDeviceMappings:
- DeviceName: /dev/sdh # Instance Store
VirtualName: ephemeral0
UserData:
Fn::Base64: !Sub
- |+
#cloud-config
write_files:
- content: |
devices:
/dev/sdb:
fs: ${FileSystem}
mountPoint: /mnt/ebs
mountOptions: ${MountOptions}
user: ubuntu
group: ubuntu
permissions: 755
label: stateful
mode: healthcheck
/dev/sdh:
fs: ${FileSystem}
mountPoint: /mnt/instance-store
mountOptions: ${MountOptions}
user: ubuntu
group: ubuntu
permissions: 755
label: ephemeral
mode: force
path: /etc/ebs-bootstrap/config.yml
bootcmd:
- curl -L -o /tmp/ebs-bootstrap "${EbsBootstrapUrlPrefix}-$(uname -m)"
- install -m755 /tmp/ebs-bootstrap /usr/local/sbin/ebs-bootstrap
runcmd:
- /usr/local/sbin/ebs-bootstrap
mounts:
- [ "LABEL=stateful", /mnt/ebs, ${FileSystem}, "${MountOptions}", "0", "2"]
- FileSystem: ext4
MountOptions: defaults,nofail,x-systemd.device-timeout=5
EbsBootstrapUrlPrefix: https://github.com/reecetech/ebs-bootstrap/releases/latest/download/ebs-bootstrap-linux
```

## Config

### `global`

#### `mode`
Assuming this is the very first launch, `ebs-bootstrap` would refuse to perform any modifications associated to the EBS device as it was assigned the `healthcheck` mode. However, we can temporarily override this behaviour with the `-mode=prompt` option. This allows the Platform Engineer to approve any suggested changes by `ebs-bootstrap`.

Specifies the mode that `ebs-bootstrap` operates in
- `healthcheck`
- Validate whether the state of a device matches its desired configuration
- Returns an exit code of `0` 🟢, if no changes are detected
- Returns an exit code of `1` 🔴, if changes are detected

### `devices[*]`

#### `fs`

The **file system** that the device has been formatted to
- If an empty string is provided, all other device properties will be ignored

#### `mount_point`

The **mount point** that the device has been mounted to
- If an empty string is provided, `owner`, `group` and `permissions` will be ignored

#### `owner`

The **user** that has been assigned ownership of the mount point
- Supports both a user **ID** and the **name** of the user

#### `group`

The **group** that has been assigned ownership of the mount point
- Supports both a group **ID** and the **name** of the group

#### `permissions`

The **permissions** that has been assigned to the mount point
- Must be specified as a three digit octal: `755`, `644`, ...

#### `label`
```
[~] sudo /usr/local/sbin/ebs-bootstrap -mode=prompt
🔵 /dev/nvme1n1: Detected Nitro-based AWS NVMe device => /dev/sdb
🔵 /dev/nvme2n1: Detected Nitro-based AWS NVMe device => /dev/sdh
🟠 Formatting larger disks can take several seconds ⌛
🟣 Would you like to format /dev/nvme1n1 to ext4? (y/n): y
⭐ Successfully formatted /dev/nvme1n1 to ext4
🟠 Certain file systems require that devices be unmounted prior to labeling
🟣 Would you like to label device /dev/nvme1n1 to 'stateful'? (y/n): y
⭐ Successfully labelled /dev/nvme1n1 to 'stateful'
...
🟣 Would you like to change ownership (1000:1000) of /mnt/ebs? (y/n): y
⭐ Successfully changed ownership (1000:1000) of /mnt/ebs
🟢 Passed all validation checks
```

The **label** assigned to the formatted device
- Labels are constrained to the limitations of the underlying file system.
- `ext4` file systems have a maximum label size of `16`
- `xfs` file systems have a maximum label size of `12`

#### `mode`
By inspecting the output of `lsblk`, we can verify that `ebs-bootstrap` was able to recover the CloudFormation assigned block device mappings (`/dev/sdb` and `/dev/sdh`) from both EBS and Instance Store NVMe devices (`/dev/nvme1n1` and `/dev/nvme2n1`) and format/label/mount the respective devices.

Provide a device-level **override** of a global `mode` property
```
[~] lsblk -o NAME,FSTYPE,MOUNTPOINT,LABEL,SIZE
NAME FSTYPE MOUNTPOINT LABEL SIZE
...
nvme0n1 8G
├─nvme0n1p1 ext4 / cloudimg-rootfs 7.9G
├─nvme0n1p14 4M
└─nvme0n1p15 vfat /boot/efi UEFI 106M
nvme1n1 ext4 /mnt/ebs stateful 10G
nvme2n1 ext4 /mnt/instance-store ephemeral 69.9G

[~] ls -la /mnt
total 16
drwxr-xr-x 4 root root 4096 Jan 8 04:57 .
drwxr-xr-x 19 root root 4096 Jan 8 04:36 ..
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 8 04:57 ebs
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 8 04:39 instance-store
```

```yaml
global:
mode: healthcheck
devices:
/dev/xvdf:
fs: "xfs"
mount_point: "/ifmx/dev/root"
owner: 1000
group: 1000
permissions: 755
label: "external-vol"
mode: healthcheck
The `mounts` module of `cloud-init` will create an entry in `/etc/fstab` for the EBS volume. The EBS volume, now labelled `stateful`, will be mounted to `/mnt/ebs` on **future reboots**. Despite device names being unstable because of the dynamical allocation behaviour of the Nitro NVMe driver, their respective labels remain stable across reboots.
```
[~] cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults,discard 0 1
LABEL=UEFI /boot/efi vfat umask=0077 0 1
LABEL=stateful /mnt/ebs ext4 defaults,nofail,x-systemd.device-timeout=5,comment=cloudconfig 0 2
```
Binary file added assets/ebs-bootstrap.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading