Adventures in bundling the spin cli with Rancher Desktop

[jandubois]

Adventures in bundling the spin cli

I recently added spin (and spin-operator) to Rancher Desktop 1.14.0. This is my feedback on the spin cli based on my notes, as it is also the first time I really worked with it, and as they say "you don't get a second chance to make a first impression".

While I have a wishlist at the very end, the bulk of my notes are just "for your information", to do with as it fits the needs of your project and available development resources.

Background

Rancher Desktop provides a comprehensive set of tools to run applications in containers or Kubernetes on the desktop as a dev environment. It works on Linux, macOS, and Windows (from WSL2 and Win32 consoles).

It comes with specific versions of the VM, the container engines, and various cli tools like docker, helm, kubectl, that have been tested and released together as a single application. Rancher Desktop is expressly not a platform package manager that will update tools at any time to the very latest version.

The bundled tools are only updated when Rancher Desktop itself is updated (generally every 2 months), but users are always able to put their own versions of tools on the PATH ahead of ~/.rd/bin (or override runtime shims, or whatever).

This is needed so that Rancher Desktop can satisfy these use-cases:

• Enterprise users can be sure when they deploy to additional workstations that each one runs the exact same (approved) versions of the software.

• New users get a predictable set of components that are pre-configured and are known to work together.

The idea of bundling spin and the spinkube operator was so that a new user could try out Spin on Kubernetes with just a few simple commands (all bundled, except npm/node):

spin new --accept-defaults --template http-js my-app
cd  my-app
npm install
spin build
spin registry push ttl.sh:my-app:1h
spin kube deploy --from ttl.sh/my-app:1h

The goal is to keep potential frustrations/failures to a minimum, to avoid the user giving up before they can experience their first app running successfully.

Of course the story above is not really complete because the user still needs to define some kind of ingress before they can view their app in the browser, but I will file this separately as a request against the spin kube plugin.

Notes

These notes will show example locations and commands from macOS, but most issues are applicable to Linux and Windows as well.

Spin maintains additional global application data

spin is not a single standalone binary, but also stores plugins and templates in a global location (~/Application Support/spin on macOS).

This presents a challenge for bundling spin because Rancher Desktop should never modify the user configuration of a different tool without the explicit consent of the user, i.e. the Rancher Desktop installer cannot simply replace an already installed plugin with a newer (or older) version.

This of course only applies to users that already have spin installed, so we decided to only install plugins and templates if none were installed already. Which means we need a way to check the status of the existing installation.

plugins and templates are treated differently

Not related to Rancher Desktop but first-time user experience:

I expected spin plugins and spin templates to work the same way, but it turned out that they don't:

Plugins

• seem to have a central catalog
• spin plugins list shows both available and installed plugins
• spin plugins install installs a single plugin, you can pick a version
• you can install and upgrade/downgrade to a specific version

Templates

• there is a default set of templates, but no catalog
• spin templates list only lists installed templates
• spin templates install installs a set of templates defined by a git repo or tarball
• you can only upgrade to "to match your current version of Spin" (but what does that mean for e.g. js-sdk templates?)

The output of the list subcommands, or the additional prompts when installing look quite different between plugins and templates. Why does spin plugins list show all available plugins, including older versions of the installed ones by default?

I'm sure there are historical reasons for the lack of symmetry, but I found it confusing initially.

Terminating the app while a prompt is waiting will not restore terminal settings

Any time the spin program displays a prompt, waiting for a single keypress, it hides the cursor, e.g.

Are you sure you want to continue? [Y/n]

If I press ⌃C instead of pressing y or n, then the program aborts, but the cursor remains invisible (I'm using iTerm2.app on macOS).

spin templates list subcommand is actively hostile to scripting/automation

You cannot use spin templates list to check if this is a fresh installation without any templates installed. It will display an interactive prompt even when stdin/stdout are not bound to a tty:

$ rm -rf ~/Library/Application\ Support/spin/
$ spin templates list </dev/null | cat
You don't have any templates yet. Would you like to install the default set? [Y/n]

This means using it in a script may hang indefinitely.

I do not consider timeout 2 spin templates list a serious option. It kind of works, but it not only hides the cursor, it also turns off echoing of stdin to the console.

Commands should not prompt for input unless stdin is attached to a tty.

list commands don't have easily parsable output

Even assuming the spin templates list command wouldn't potentially hang when there are no templates installed, the output is also not easily parsable by another process.

Many cli applications offer a --json, --format …, or --template … option to provide more control over how the information is presented. This allows automation/scripting to treat the output like an API instead of having to parse output generated for human consumption, which might also not be stable from release to release. For example:

$ docker images --format=json | jq -r .Repository
quay.io/jetstack/cert-manager-startupapicheck
quay.io/jetstack/cert-manager-controller
…

spin doesn't seem to support this. I've hacked up a prototype for spin templates list --json. It turns all spin-template.toml files into JSON format, adds a dir field to each entry, and concatenates them:

$ SPIN_APPDATA="$HOME/Library/Application Support/spin"
$ spin_templates_list_json() {
>   find "$SPIN_APPDATA/templates" -depth 1 -type d -exec \
>      yq -o=json -I=0 '.dir="{}"' "{}/metadata/spin-template.toml" \;
> }

Once the data is available as an API users can build their own tools on top of it without having to write a whole plugin. Examples:

$ # How many templates are installed?
$ spin_templates_list_json | jq --slurp length
12

$ # What is the install directory of the http-c template?
$ spin_templates_list_json | jq -r 'select(.id == "http-c").dir'
/Users/jan/Library/Application Support/spin/templates/http-c_c328011daf311449f512a99cc0e60bbcae479c2b9d0792609b69834a0aec067f

$ # Show a table of template ids and descriptions
$ (echo $'ID\tDESCRIPTION'; spin_templates_list_json | jq -r '[.id, .description] | @tsv') | column -s$'\t' -t
ID                 DESCRIPTION
static-fileserver  Serves static files from an asset directory
redirect           Redirects a HTTP route
http-rust          HTTP request handler using Rust
http-swift         HTTP request handler using SwiftWasm
…

While it doesn't really matter in this particular case, I think using a streaming "newline delimited JSON" format like I've done here is a better choice than wrapping all objects in a useless array. docker … --format json produces the same format, and as shown above, it works well with jq.

I agree that maybe there are not that many compelling use-cases for accessing template and plugin metadata yet, but I think it will become useful for your own integration tests and should be simple to implement.

Personally I think every (sub)command/plugin that queries the state of a system should have a --json option to provide the results as an API. You can't predict what people want to build on top of your tools.

Where does spin store the plugins and templates

Ideally I wouldn't need to know, but since I can't use the cli to query if templates and plugins have been installed or not, I needed to check the filesystem.

I could not find a way to find the directory from the command itself, like I can do with e.g. docker:

$ docker info -f json | jq -r .DockerRootDir
/var/lib/docker

It would be nice to have a similar spin info subcommand that provides the data directory location, e.g.

$ spin info --json | jq -r .SpinAppData
/Users/jan/Library/Application Support/spin

The rules for locating the data directory are complex

Since I can't query the spin app data location, I have to hard-code the rules from the spin app into the installer:

• Linux: $HOME/.local/share/spin
• macOS: $HOME/Application Support/spin
• Windows: %LOCALAPPDATA%\spin

Unfortunately it gets more complicated: On Linux it will be $XDG_DATA_HOME/spin if XDG_DATA_HOME is defined. #1494 makes the location dependent on installation by Homebrew, and #641 promises to make things even more complex.

I think replicating all these rules in a 3rd-party installer is bad, but it gets even worse when you consider GUI applications. They are not started from the user's shell, but e.g. on macOS from launchd on behalf of Finder.app. Those apps don't inherit the user's shell environment variables. You could try to look up the default shell for the user from /etc/passwd and then run a login shell as a subprocess to query the environment variables, but this is getting just too convoluted, and is still not guaranteed to be correct 100% of the time either.

Is there any override to specify the location of the data directory?

It would be great if we could set an environment variable, e.g. SPIN_APPDATA, and it would take precedence over dirs::data_local_dir() inside spin.

Then Rancher Desktop could just create a simple wrapper in ~/.rd/bin/spin:

export SPIN_APPDATA="$HOME/Library/Application Support/rancher-desktop/spin"
exec "/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/bin/spin" "$@"

This way the directory would be "owned" by Rancher Desktop and could be updated with new plugin and template versions whenever Rancher Desktop is updated.

An advanced user could still install spin in a directory that is on the PATH in front of ~/.rd/bin and manage the spin version, plugins, and templates themselves, and it would never be touched by Rancher Desktop. Best of both worlds!

You can't install the default templates unless git is installed

$ ./spin templates install --git https://github.com/fermyon/spin
Copying remote template source
Error: Failed to install one or more templates

Caused by:
   0: Failed to get template source
   1: Error cloning Git repo https://github.com/fermyon/spin: `git` command not found - is git installed?

There is no way to install templates from a remote tarball

Since we can't rely on git being installed, we need to install using --dir from a local directory because there is no option to install from a remote tarball like this:

spin templates install --from https://github.com/fermyon/spin/archive/refs/heads/main.tar.gz

That means we need to download the file ourselves, unpack it into a temporary directory, install from there, and then remove the temporary directory again.

Since we are doing this from a shell script, it means we need curl or wget, neither of which are guaranteed to be installed.

This problem can be avoided by moving the installation code from the shell script into the Electron app and downloading the tarball with node-fetch. For now, we skip installing templates if we don't find curl or wget.

spin.exe (and plugins written in Rust) depend on the proprietary vcruntime140.dll

spin.exe on Windows is dynamically linking to vcruntime140.dll, a proprietary C runtime library from Microsoft that is not part of Windows itself. This is already discussed in #1504.

While there are likely ways to legally distribute the additional DLL, this would require approval from corporate lawyers. All SUSE software is released as free and open source software, so bundling a proprietary closed source library is not really possible, as the package as a whole would no longer be available for redistribution under the OSS license.

This issue is due to the way Rust binaries are built by default for Windows systems. It is possible to statically link the runtime library and avoid this issue.

Plugins written in Rust (e.g. js2wasm) have the same issue, while plugins written in Go (e.g. kube) do not because Go produces statically linked binaries by default.

Note that Windows Update will not automatically update the vcruntime140.dll when installed with another software project; it will only be updated when installed using the Microsoft installer into the Windows system locations.

So bundling vcruntime140.dll with the app itself does not help with getting bug fixes for the runtime earlier; they would still have to be installed with a spin (or Rancher Desktop) update. The only disadvantage of static linking would be slightly larger executables.

static-linux executable installs dynamically linked plugins

The dynamically linked Linux version of spin doesn't work on Alpine Linux because of glibc dependencies (as expected). Thankfully there is a static-linux version:

$ wget https://github.com/fermyon/spin/releases/download/v2.5.1/spin-v2.5.1-static-linux-amd64.tar.gz
…
$ tar xfz spin-v2.5.1-static-linux-amd64.tar.gz
$ ldd spin
	/lib/ld-musl-x86_64.so.1 (0x7f8c09e4f000)

Unfortunately spin plugin install does not in turn fetch a "static-linux" version of the plugin (maybe they don't exist?), but installs a dynamically linked Linux one, which doesn't work:

$ ./spin plugins install js2wasm
…
Plugin 'js2wasm' was installed successfully!
…

$ PATH="$PWD:$PATH" spin js2wasm --help
Error: No such file or directory (os error 2)

$ ldd .local/share/spin/plugins/js2wasm/js2wasm >/dev/null
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by .local/share/spin/plugins/js2wasm/js2wasm)
Error relocating .local/share/spin/plugins/js2wasm/js2wasm: __snprintf_chk: symbol not found
Error relocating .local/share/spin/plugins/js2wasm/js2wasm: __vsnprintf_chk: symbol not found
Error relocating .local/share/spin/plugins/js2wasm/js2wasm: __register_atfork: symbol not found
Error relocating .local/share/spin/plugins/js2wasm/js2wasm: gnu_get_libc_version: symbol not found

I think static-linux needs to be treated as a separate platform, and plugins need to be built/published for it.

spin build launches spin via a PATH search

I noticed that spin build tries to run spin subcommands (maybe to run the js2wasm plugin), and relies on the PATH to find it:

$ ./spin build
Building component my-app with `npm run build`
…
webpack 5.91.0 compiled successfully in 88 ms
sh: spin: command not found
Error: Build command for component my-app failed with status Exited(127)

It works correctly when spin is on the PATH:

$ PATH="$PWD:$PATH" ./spin build
Building component my-app with `npm run build`
…
Finished building all Spin components

This still doesn't work if you rename spin to e.g. spin-2.5.1 when e.g. you have multiple versions installed for testing. The subprocess may invoke a different spin than the one running in the toplevel process.

I can't think of a scenario where the current behaviour is an advantage and would rather use std::env::current_exe() instead of a hard-coded name.

Executables are not signed

I'm not talking about cosign, but about the "normal" code signing that is being checked by AppLocker on Windows or GateKeeper on macOS.

This is kind of expected, as GHA typically don't have access to corporate signing keys. Rancher Desktop releases are also signed offline and then uploaded again to GitHub releases.

We will consider signing even third-party binaries that we bundle to allow users with restrictive security profiles to run these binaries.

It turns out that we accidentally already signed the spin binary in the Rancher Desktop 1.14 release on macOS (but not Windows). Unfortunately signing the binary seems to break the spin up functionality to run applications locally (outside containers). Tracked in #2553. Update: spin needs the com.apple.security.cs.allow-unsigned-executable-memory entitlement when using the hardened runtime (required for notarization). Will be fixed in Rancher Desktop 1.14.2.

There are no Windows on ARM binaries

We are not yet shipping Rancher Desktop for Windows on ARM either, but I've seen the steps ARM (the company) goes through to build their own version in-house, and would like to make things easier. It will also be needed when we eventually decide to support this platform directly.

I'm trying to encourage all our 3rd-party dependencies to provide Windows on ARM binaries, so we can update our build scripts to use them automatically instead of requiring people to build them themselves.

For Go applications this is typically just the addition of a cross-compilation target to the Makefile. I don't know what would be required for Rust, and what this means for statically linking the runtime.

Wishlist

The following changes would be helpful for future releases of Rancher Desktop. Everything else above does not really affect us (or can be worked around), and is only provided as feedback for you.

I'm listing them in the order of priority from the Rancher Desktop project, but understand that this might not match the interest/priorities of the spin project.

1. Support for SPIN_APPDATA to override dirs::data_local_dir()
2. Don't break spin up when the executable is signed (on macOS)
3. Statically linked spin.exe and plugins for Windows
4. Statically linked plugins for Linux (automatically installed by spin-static-linux)
5. Binaries for Windows on ARM (low priority right now, but eventually desired)

If you create any actual GitHub issues based on any of the topics of this document, please reference @jandubois in the description, so I can subscribe and keep track of changes. Or let me know if I should create separate issues for entries from the wishlist!

[lann]

Closing the loop on a couple of the "wishlist" items here; it appears that GitHub Discussions don't populate backlinks for references:

1. Support for SPIN_APPDATA to override dirs::data_local_dir()

• Support overriding the "Spin data dir" via env var #2563

2. Don't break spin up when the executable is signed (on macOS)

• Signing the spin executable breaks it #2553

[jandubois]

Closing the loop on a couple of the "wishlist" items here; it appears that GitHub Discussions don't populate backlinks for references:

Thanks!

1. Support for SPIN_APPDATA to override dirs::data_local_dir()

• Support overriding the "Spin data dir" via env var #2563

Thanks for adding this! Rancher Desktop 1.17 will be using this feature to maintain its own data directory for spin and force the plugin and template versions to the ones pinned in the release. This will hopefully prevent any further breakage from plugin updates that are not backwards compatible. Users are alway free to put their own version of utilities earlier in the PATH if they want to manage them themselves.

2. Don't break spin up when the executable is signed (on macOS)

• Signing the spin executable breaks it #2553

Not sure what you have done, but this is no longer an issue since Rancher Desktop added the required entitlements for the hardened runtime automatically.

[itowlson]

spin templates list </dev/null | cat

The interaction happens on stderr. spin templates list 2>/dev/null | cat works correctly in both the "installed" and "not installed" cases. Is that sufficient, or do you need it to suppress the problem in the stdout case as well?

[itowlson]

I noticed that spin build tries to run spin subcommands (maybe to run the js2wasm plugin)

I think this was only an issue for the old JavaScript and Python templates. These now invoke componentize-js (installed as part of npm install) and componentize-py (installed via pip3 install). I can look into detecting when spin build is invoking Spin if the older templates are still a concern for you, but we'd strongly encourage users to move to the updated templates. (And I understand this would still be a consideration for you if a future template depended on a plugin.)

[itowlson]

There is no way to install templates from a remote tarball

Raised #2955 for this and working on it today.

[itowlson]

list commands don't have easily parsable output

spin templates list --json exists; I've raised #2956 for spin plugins list --json.

[itowlson]

Raised #2965 for Windows ARM binaries.

[itowlson]

Raised #2961 for statically linked plugins (needs discussion)

[itowlson]

@jandubois Circling back to this to sync on where we are at as Spin 3.1 goes to the CD mastering plant.

Outstanding items in the priority list:

• Statically linked plugins for Linux (automatically installed by spin-static-linux)
  • I wanted to check in on this. Given the changes by which JS and Python no longer require plugins, is this still a high priority, or can we put it in the "the right thing to do but not a blocker" bucket? Asking because this seems like it's going to need more pondering...
• Windows ARM binaries - you've indicated low demand for this for now, but we understand it remains desirable to have

Of the other items in the priority list:

• Data directory environment variable and signing fixes are already in
• Statically linking Windows vcruntime will be in 3.1

Issues raised but not in the priority list:

• spin templates list --json is already in, spin plugins list --json will be in 3.1
• Install templates from remote tarball will be in 3.1
• "Find the location of the data directories" should no longer be needed with the other improvements
• spin build launches plugins via the Spin on PATH should no longer be an issue with js2wasm and py2wasm going away

Things which are not looking likely to be fixed:

• Restoring the cursor is a third party library issue which it's tricky for us to work around. I've looked into it but not yet found a satisfactory solution.

Does this line up with your understanding, and is it enough to unblock you / largely reduce your pain?

[jandubois]

Outstanding items in the priority list:

• Statically linked plugins for Linux (automatically installed by spin-static-linux)

  • I wanted to check in on this. Given the changes by which JS and Python no longer require plugins, is this still a high priority, or can we put it in the "the right thing to do but not a blocker" bucket? Asking because this seems like it's going to need more pondering...

Yes, doesn't matter to us. We install js2wasm and kube in 1.17 but will only install kube in 1.18, so it won't affect us.

• Windows ARM binaries - you've indicated low demand for this for now, but we understand it remains desirable to have

I think so, but we'll see. Windows on ARM seems perpetually in the future, but doesn't really get closer.

Of the other items in the priority list:

• Data directory environment variable and signing fixes are already in

Yes, thank you!

• Statically linking Windows vcruntime will be in 3.1

Good! We have 3.0 in the 1.17 release, but will pick up 3.1 automatically when it is released.

Issues raised but not in the priority list:

• spin templates list --json is already in, spin plugins list --json will be in 3.1

This does not matter to us, as we worked around it, and now use SPIN_DATA_DIR and just reinstall plugins and templates on each start.

• Install templates from remote tarball will be in 3.1

This will allow us to simplify the spin installer code.

• "Find the location of the data directories" should no longer be needed with the other improvements

+1

• spin build launches plugins via the Spin on PATH should no longer be an issue with js2wasm and py2wasm going away

+1

Things which are not looking likely to be fixed:

• Restoring the cursor is a third party library issue which it's tricky for us to work around. I've looked into it but not yet found a satisfactory solution.

That's unfortunate, as I have a habit of using ⌃C to abort prompts. Can't you setup a signal handler that will restore the cursor and terminal echo?

Does this line up with your understanding, and is it enough to unblock you / largely reduce your pain?

Yes, I think we are all good now, thank you!

I've sent feedback elsewhere about my dismay in how the incompatible changes in spinkube and the spin plugin have been handled, but by forcefully reinstalling the matching versions on each start we hopefully have isolated ourselves in case this is going to happen again.

[itowlson]

Can't you setup a signal handler that will restore the cursor

We did give this a try, but ran into some challenges: see discussion in #2559. There's an issue on the upstream crate at console-rs/dialoguer#294. I completely sympathise with everyone's frustration about this!

[jandubois]

We already use a wrapper to setup SPIN_DATA_DIR. Maybe I should see if we can just call the real spin instead of using exec and somehow keep running even when spin receives a SIGINT.

Then we could finish the wrapper with tput cnorm; stty echo and all would be good. Maybe all we need is a trap … INT, wait for the child to exit, and then clean up the terminal.

Anyways, low priority, as this is not "our" issue... 😄