[feature]: brainstorm: OS-level packaging for lnd (brew, apt, etc)

[Roasbeef]

Opening this as a brainstorm rather than a proposal. I don't think we know yet whether we want to be in the OS packaging business at all, but the question keeps coming up, and I'd rather have the design space and the sharp edges written down somewhere than re-derive them in a Slack thread every six months.

The rough shape of the idea: what would it take for a user to run brew install lnd or apt install lnd, and then pick up new releases with a plain brew upgrade / apt upgrade? Today the only supported paths are the release tarballs, building from source, and Docker. That's workable for people who run nodes professionally, but it means every routine version bump is a manual, multi-step ritual: download, import keys, verify a manifest, copy binaries over the old ones, restart, unlock. Lots of places to get it subtly wrong.

The concrete version of that complaint, from people running nodes out in the wild: it's hard to motivate operators to upgrade at all. The ritual is enough friction that nodes just sit on old releases, and that's its own kind of risk, especially for the releases carrying security fixes. "Upgrading is annoying" is a weaker argument for doing this than "a meaningful chunk of the network is running stale code because upgrading is annoying", and I think it's actually the second one.

For context on the current state of the world, the packages that exist are all third party. nixpkgs carries lnd (currently 0.21.2-beta, built with buildGoModule, copying RELEASE_TAGS verbatim out of make/release_flags.mk), and there's an AUR lnd maintained by a community member. There is no lnd formula in homebrew-core and no .deb anywhere, so the two most common platforms have nothing.

What's already in the tree

Part of why this feels worth thinking about now is that a chunk of the input side already exists:

• scripts/release.sh builds reproducible, -trimpath'd binaries for 15 platforms and emits a manifest-<tag>.txt of sha256s.
• scripts/verify-install.sh enforces MIN_REQUIRED_SIGNATURES=5 against the 16 maintainer keys in scripts/keys/. That 5-of-N multisig over the manifest is our real trust root, and it's stronger than what any distro offers.
• scripts/gen_man_pages.sh already honors DESTDIR and PREFIX, which happens to be exactly the interface a .deb/.rpm build wants.
• contrib/lncli.bash-completion and contrib/init/lnd.service exist.
• .github/workflows/verify-release.yaml already fires on release: published and re-verifies signatures.

So the marginal work isn't "build binaries for N platforms", it's packaging, hosting, and (the interesting part) deciding what an upgrade should actually do to a running node.

The fork in the road: rebuild vs repackage

Most of the other questions seem to hang off this one.

Every third-party package today rebuilds from source. That's how nixpkgs and the AUR work, and it's what homebrew-core would require of us. The downside is that rebuilding severs the chain back to the 5-signature manifest: the user ends up trusting the packager and the distro build infra rather than our maintainers. It's also a quiet way for build tags to drift out of sync with RELEASE_TAGS, so someone could end up with an lnd missing kvdb_postgres or signrpc without realizing it.

The alternative is repackaging the binaries we already build and sign, so the .deb becomes a thin wrapper whose contents hash-match a manifest five maintainers signed. We could even ship manifest-<tag>.txt and the .sig files inside the package (somewhere like /usr/share/lnd/) so it's re-verifiable after install.

That second option has a property I find appealing: it makes any APT repo signing key a transport key rather than a root of trust. A repo key has to live in CI, so it can't be one of the maintainer release keys, and it'd be unfortunate if an automation key were the only thing standing between a user and a bad binary. Under repackaging it isn't. But I could be talked out of this; there's a real argument that meeting distros where they are (source builds) is better for adoption even if it's worse for provenance.

Homebrew

If we did this, a tap (lightninglabs/homebrew-lnd) looks a lot more tractable than homebrew-core.

homebrew-core seems closed to us on two counts. Their acceptable formulae policy says "Upstream must identify the packaged version as stable and provide an immutable tag or release", and every tag we cut is -beta. It also requires that a formula "must either build from source or install portable, platform-independent output", which drops us back into the rebuild tradeoff above. Someone should sanity check whether the -beta thing is actually enforced in practice, but on paper it's a blocker.

A tap has neither constraint. It could install the prebuilt darwin-arm64/amd64 and linux-amd64/arm64 tarballs, verify sha256 against the manifest, and give people brew install lightninglabs/lnd/lnd plus ordinary brew upgrade. Of everything in this issue this looks like the smallest piece by a wide margin: a formula, plus a CI job that opens a bump PR on release, and then close to zero ongoing cost.

apt and dnf

The build side seems mostly solved by nfpm, which produces deb, rpm, and apk from one YAML file. It'd wrap the binaries, the generated man pages, the completion script, the systemd unit, and sample-lnd.conf installed as a dpkg conffile at /etc/lnd/lnd.conf.

Hosting is the open one, and the first fork there is whether we go through a Launchpad PPA or run a repo ourselves. The PPA is the low effort answer and I think it's the wrong one, for a harder reason than just preferring our own infra: Launchpad won't accept binary uploads at all. Mixed source/binary uploads are rejected by design, so that users can always see the source for what they're running, which means a PPA would build lnd on Canonical's builders out of a source package. That quietly picks the rebuild branch of the fork above and severs the chain back to the signed manifest, and it's Ubuntu only besides. So if the repackaging argument holds, a PPA isn't really available to us, it just looks like it is.

That leaves a repo we run: aptly against S3 + CloudFront, versus something hosted like Cloudsmith. Self-hosting is more control and more key ceremony; hosted is faster to stand up and adds a vendor to the trust path. My lean is our own infra, for the same reason as above, the fewer parties between a maintainer signature and the operator's disk the better. Whichever way, we'd presumably want at least stable and rc suites so RC testers can opt in without release candidates landing on mainnet nodes by accident.

The ongoing maintenance cost here is the thing I'm least sure about. A package repo is infrastructure that has to keep working, and it's a new thing to be on the hook for.

Things that look like dead ends

Debian proper doesn't seem realistic. Debian policy forbids vendored dependencies, so every one of lnd's hundreds of Go deps would need its own golang-github-* source package, plus a Debian Developer willing to sponsor and a full DFSG license review. That reads like a multi-year effort whose payoff is a permanently stale lnd in stable, which is arguably worse than no package.

Snap is a harder no, and for a reason that has nothing to do with packaging mechanics. Snaps auto-refresh by default, four times a day, and the refresh is driven by the store rather than by the operator. You can now hold that off indefinitely (snap refresh --hold=forever, since snapd 2.58), but it's an opt out that the operator has to know exists, on every machine, and the default is still that the daemon updates itself. Put that next to the migration point below: a snap that auto-refreshes into a release carrying a channel DB migration has silently taken away the ability to roll back, on someone's node, without them asking for it. Whatever we end up doing here, the upgrade should be pulled. apt update && apt install lnd, when the operator decides, not on Canonical's timer.

The verification story doesn't help either. In the snap model the binding between a snap name and its content hash is the snap-revision assertion, and that's signed by Canonical as the store authority, not by us. Publishers do get keys, but snap-build assertions, the ones that would attest that we built this, are documented as traceability only and aren't checked by snapd. So there's nowhere to hang a 5-of-N maintainer signature over the artifact, which is the property in this whole design I'd least like to give up. An APT repo at least has a PGP key the operator explicitly authorized sitting in the path, and under the repackaging option above that key is only a transport key in front of a manifest five maintainers signed.

Flatpak falls out for a plainer reason: strict confinement fights with what lnd needs to do, reach a local bitcoind over a socket or ZMQ, read a user-chosen data dir, etc, etc. By the time you've punched enough holes in the confinement it's not clear what you bought. That objection applies to snap's confinement too, on top of everything above.

Happy to be wrong on any of these if someone has actually shipped through them.

The part that worries me

Packaging a daemon that custodies funds is not the same problem as packaging jq, and I think this, not the nfpm config, is where the real work and real risk live. apt upgrade is designed to restart services. For us that cuts a few ways:

Migrations are one way. This is the one that actually scares me. Someone running unattended-upgrades gets a new lnd, gets their channel DB migrated on next start, and then can't roll back. There's no undo. Maybe that means a postinst that never restarts by default, maybe it means deliberately staying out of the default unattended-upgrades allowlist, maybe it means the packaging has to surface release notes for migration-carrying versions somehow. I don't have a good answer and I'd like other opinions.

Wallet unlock. A package-driven restart leaves the node locked and offline unless wallet-unlock-password-file is set. Silent downtime is exactly the failure people don't notice until a channel force closes.

Data directory and system user. A package that creates an lnd system user with /var/lib/lnd orphans everyone's existing ~/.lnd. So either we document a migration carefully, or the package ships binaries plus a unit file and creates nothing.

Separately, while poking at this I noticed contrib/init/lnd.service isn't quite packaging-ready as written. It hardcodes /usr/local/bin/lnd where a deb would install to /usr/bin/lnd, and its Requires=bitcoind.service will hard-fail startup on a neutrino node with no local bitcoind at all. Probably wants to be Wants= plus After=, or dropped. That one seems worth fixing on its own regardless of whether any of this happens.

One CI gotcha

If anyone does prototype this: packaging would need to chain off release: published, not off the tag push.

release.yaml fires on push: tags and creates a draft release, and the manifest signatures get collected out of band afterward by the maintainers. A tag-triggered packaging job would cheerfully build and publish packages that predate the signatures they're supposedly derived from. The natural hook looks like a job running after verify-release.yaml succeeds, so packages can only ship once five signatures verify.

Open questions

Mostly writing these down to argue about:

• Do we even want to own this? Every package we ship is a support surface, and "my node broke after apt upgrade" becomes our problem in a way it currently isn't. The counterweight is the upgrade-rate problem up top: nodes sitting on stale releases are also our problem, just a quieter one that never files an issue.
• Binaries only (no system user, no auto-restart, user manages their own data dir and unit) or full service management (system user, /var/lib/lnd, managed restarts)? The first is far safer and far less work. The second is usually what people mean when they ask for apt install lnd, and getting a working lnd.service out of the install is a good chunk of what makes a package worth more than a tarball. Those two aren't quite the same choice though: we can ship the unit file without enabling or starting it, so the operator gets a correct systemd unit for free and still decides when the daemon comes up. That middle path is where I'd start.
• Is a brew tap alone enough to cover most of the actual pain, given how much of the "I want to upgrade easily" crowd is on macOS?
• Would we rather invest in the third-party packages that already exist, adopting the AUR package and filing nixpkgs bumps ourselves, instead of standing up new infrastructure?
• Is there prior art worth copying here? bitcoind notably does not ship a first-party apt repo, which might be telling.

No strong conclusions from me. Curious what others think, especially on the migration-safety question and on whether the ongoing maintenance is worth it.

[Roasbeef]

Folded some team feedback into the issue body. Summarizing what changed here so anyone who already read it doesn't have to diff it.

The strongest note was on snap, and it's a better objection than the one I'd written. I had snap and Flatpak lumped together as "confinement is a poor fit", which is true but beside the point. The real problem is the update model: snaps auto-refresh by default, four times a day, driven by the store rather than by the operator. That's the opposite of what you want for a daemon whose upgrades carry one-way channel DB migrations. An operator can wake up to a node that refreshed itself into a release it can no longer roll back out of, having never asked for it. To be fair to snapd, snap refresh --hold=forever has been there since 2.58, so the "you can't turn it off" version of the complaint isn't literally true anymore. But it's an opt out you have to know about, per machine, and the default still has the daemon updating itself. apt gets this right by construction: apt update && apt install lnd happens when the operator decides.

The verification angle turns out to cut the same way, and I hadn't looked at it closely before. In the snap model the binding from snap name to content hash is the snap-revision assertion, and Canonical signs that as the store authority, not us. Publishers do get keys, but snap-build assertions (the ones that would attest that we built the thing) are documented as traceability only and snapd doesn't check them. So there's nowhere in that model to hang a 5-of-N maintainer signature, which is the property I'd least like to trade away. An APT repo at least puts a PGP key the operator explicitly authorized in the path, and under the repackaging option that key is only a transport key sitting in front of a manifest five maintainers signed.

On hosting, the suggestion was a repo we run rather than a PPA, on the general principle of keeping things in our own infra. Agreed, and it turns out there's a hard technical reason underneath the preference: Launchpad rejects binary uploads outright. Mixed source/binary uploads are refused by design so users can always see the source for what they're running, which means a PPA necessarily builds lnd on Canonical's builders from a source package. That silently picks the rebuild branch of the fork above and cuts the chain back to the signed manifest, and it's Ubuntu-only besides. So the PPA option isn't really on the table if we care about repackaging, it just looks like it is. I've written that up in the apt section.

Last one: shipping a systemd unit in the install was called out as a concrete win, and I think it partly dissolves the binaries-only vs full-service-management question I'd posed as a binary. Those aren't the same choice. We can ship a correct lnd.service in the package without enabling or starting it, so the operator gets the unit for free and still decides when the daemon comes up. That's the narrow start, and it sidesteps the restart-on-upgrade hazard entirely.

Also worth recording the motivation, since it's better than the one I opened with: the complaint from people running nodes isn't "upgrading is annoying", it's that operators can't be motivated to upgrade at all, so nodes sit on stale releases. That's a real argument for doing something here, and it's the counterweight to the support-surface concern. A node stuck three releases back never files an issue.

Still no strong conclusion from me on whether we take this on. But the shape is clearer: if we do it, it's a repo we run, serving repackaged binaries that hash-match the signed manifest, with pulled upgrades and a unit file we install but don't start.

[GustavoStingelin]

Some useful prior art: Floresta already has a formula in Homebrew Core. The formula builds florestad and floresta-cli from a tagged source archive, publishes bottles for macOS and Linux, and provides an optional service definition for florestad.

[GustavoStingelin]

Did you evaluate https://github.com/goreleaser/goreleaser too, or just https://github.com/goreleaser/nfpm?

[ZZiigguurraatt]

Someone running unattended-upgrades gets a new lnd, gets their channel DB migrated on next start, and then can't roll back. There's no undo.

Can we recommend people uninstall unattended-upgrades that are using lnd?

[ZZiigguurraatt]

On hosting, the suggestion was a repo we run rather than a PPA, on the general principle of keeping things in our own infra. Agreed, and it turns out there's a hard technical reason underneath the preference: Launchpad rejects binary uploads outright. Mixed source/binary uploads are refused by design so users can always see the source for what they're running, which means a PPA necessarily builds lnd on Canonical's builders from a source package. That silently picks the rebuild branch of the fork above and cuts the chain back to the signed manifest, and it's Ubuntu-only besides. So the PPA option isn't really on the table if we care about repackaging, it just looks like it is. I've written that up in the apt section.

As an analogy, Zeus has their own F-Droid repo for similar reasons: https://zeusln.com/download/fdroid

[ZZiigguurraatt]

Don't think it was mentioned above, but if we do this, we likely will want to also bundle bitcoind along with lnd in the deb package, or make a separate deb package for bitcoind and make lnd dependent on it.

[ZZiigguurraatt]

Separately, while poking at this I noticed contrib/init/lnd.service isn't quite packaging-ready as written. It hardcodes /usr/local/bin/lnd where a deb would install to /usr/bin/lnd, and its Requires=bitcoind.service will hard-fail startup on a neutrino node with no local bitcoind at all. Probably wants to be Wants= plus After=, or dropped. That one seems worth fixing on its own regardless of whether any of this happens.

+1 on this