Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Build debian packages and run autopackage tests #130

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

3v1n0
Copy link
Collaborator

@3v1n0 3v1n0 commented Dec 5, 2023

Add a workflow to build debian packages and run autopkgtests.

Requires: canonical/desktop-engineering#58

UDENG-2343

@3v1n0 3v1n0 requested a review from a team as a code owner December 5, 2023 21:22
@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.18%. Comparing base (36511cd) to head (dee7b3a).
Report is 73 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
- Coverage   83.43%   83.18%   -0.26%     
==========================================
  Files          83       87       +4     
  Lines        8689     8771      +82     
  Branches       74       74              
==========================================
+ Hits         7250     7296      +46     
- Misses       1111     1143      +32     
- Partials      328      332       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@3v1n0 3v1n0 changed the base branch from debian-packaging to main December 5, 2023 21:30
@3v1n0 3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 9df4ea2 to 7281cd3 Compare December 6, 2023 02:09
Copy link
Member

@didrocks didrocks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of assumptions on that PR, in particular in which distro to runs. Also, the containers in polluted with preparing the build package and end up having network connection when building it, which is not what a real ppa or builder is doing.

I think it will be better to reuse https://github.com/canonical/desktop-engineering/blob/main/gh-actions/common/build-debian/action.yml, amend it for anything needed without extending too much complexity. Please check with @EduardGomezEscandell who is the primary author.

Some of the changes that can be made for instance it to not rely on

    - name: Build package
      uses: jtdor/build-deb-action@v1

For instance, which should be easy enough to do.

That way, we can decide on which releases to build and ensure we have something closer to a real build environement. Adding autopkgtests as a separate action is interesting there too.

.github/workflows/build-deb.yaml Outdated Show resolved Hide resolved
.github/workflows/build-deb.yaml Show resolved Hide resolved
@3v1n0
Copy link
Collaborator Author

3v1n0 commented Dec 11, 2023

There is a lot of assumptions on that PR, in particular in which distro to runs.

Consider this was done as a quick thing to test #120 (and actually caught few issues during the final testing of that PR :)).

So, there are assumptions but all depending on variables, so ideally can be multiplexed to multiple jobs, although this was not a requirement yet.

Also, the containers in polluted with preparing the build package and end up having network connection when building it, which is not what a real ppa or builder is doing.

Yeah, this has to be changed indeed.

I think it will be better to reuse https://github.com/canonical/desktop-engineering/blob/main/gh-actions/common/build-debian/action.yml, amend it for anything needed without extending too much complexity. Please check with @EduardGomezEscandell who is the primary author.

However I was looking at the code of that action and I'm not sure how internet is prevented there too, since it seems that certificates are still installed. However I'll dig on that more further.

@EduardGomezEscandell
Copy link

EduardGomezEscandell commented Dec 12, 2023

Hi, thanks for taking a look :)

However I was looking at the code of that action and I'm not sure how internet is prevented there too, since it seems that certificates are still installed. However I'll dig on that more further.

The action works in two phases, each in a separate container.

  • The first one builds the source package, with certificates installed
  • The second one builds the package, with no certificates.

@3v1n0
Copy link
Collaborator Author

3v1n0 commented Dec 12, 2023

  • name: Build package
    uses: jtdor/build-deb-action@v1

Something I don't like of that is that it's running the build process as root, which is what other builders do, but it should not be the case. I can probably patch that to fix this though.

@3v1n0
Copy link
Collaborator Author

3v1n0 commented Feb 25, 2024

I've tried again using canonical/desktop-engineering/gh-actions/common/build-debian but still this doesn't work as per this error:

  dh_auto_configure --buildsystem=cargo
  debian cargo wrapper: options, profiles, parallel, lto: ['parallel=4'] ['noudeb'] ['-j4'] 0
  debian cargo wrapper: rust_type, gnu_type: x86_64-unknown-linux-gnu, x86_64-linux-gnu
  error: current package believes it's in a workspace when it's not:
  current:   /github/workspace/tmp.Vj6UHjMoxL/source/vendor_rust/tokio/Cargo.toml
  workspace: /github/workspace/Cargo.toml
  this may be fixable by adding `tmp.Vj6UHjMoxL/source/vendor_rust/tokio` to the `workspace.members` array of the manifest located at: /github/workspace/Cargo.toml
  Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
  malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/cargo/bin/dh-cargo-vendored-sources line 23.
  dh_auto_configure: error: /usr/share/cargo/bin/dh-cargo-vendored-sources returned exit code 255
  make[1]: Leaving directory '/github/workspace/tmp.Vj6UHjMoxL/source'
  make[1]: *** [debian/rules:68: override_dh_auto_configure] Error 25
  make: *** [debian/rules:50: binary] Error 2
  dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

It's quite weird since it does seems to run in the same environment.... And funny thing: there's no relevant difference in source packages as per debdiff.

So.... I'm not really sure what it is. I'll check in future, but I think that for now the safest way would just go with this option (once I address the comments on building based on the source and disabling internet access), instead of continuing living without a CI action that ensures we're still packaging authd properly.

@3v1n0
Copy link
Collaborator Author

3v1n0 commented Feb 26, 2024

Opening this again, even though now it's using canonical/desktop-engineering#28

.github/workflows/build-deb.yaml Outdated Show resolved Hide resolved
3v1n0 added a commit to canonical/desktop-engineering that referenced this pull request Nov 13, 2024
…s and add tests (#28)

As underlined in
ubuntu/authd#130 (comment) using
`jtdor/build-deb-action` for building debian packages has some problems
that are yet not clear, and it also does not seem to properly support
building non-native packages.

As per this, given that the code needed for handling the build inside a
simpler docker container is quite small, I've moved the build to manual
handling which allows us some more control over the steps at the cost of
being a bit more verbose.

As bonus:
- We now only build as user (not the sources, since there's no much
point)
- All the network traffic is totally blocked for the builder user (not
just HTTPS)
- ~~`DEB_BUILD_OPTIONS` can be customized for special builds (e.g.
`nocheck`)~~
 - Version uses actual distro version

I've added some commits from #22, but not included the lintian build yet
(that I've ready
[here](3v1n0@1b72205))
not to make this PR harder to check, but also because I think those
could instead be part of another action to allow more parallelization
when used.

By doing this change, not only both test cases pass, but also authd is
correctly built: ubuntu/authd#130 (e.g.
https://github.com/ubuntu/authd/actions/runs/8044158864)

UDENG-2439
@3v1n0 3v1n0 marked this pull request as draft December 13, 2024 16:30
@3v1n0 3v1n0 marked this pull request as draft December 13, 2024 16:30
@3v1n0 3v1n0 force-pushed the debian-packaging branch 2 times, most recently from 95bab82 to 032a1f9 Compare December 14, 2024 02:18
3v1n0 added a commit to canonical/desktop-engineering that referenced this pull request Dec 18, 2024
…uild-source phase (#54)

When building some packages we may need to prepare the build environment
before preparing the source package, for example in some cases we may
need to enable git access (to compute the package version or similar) or
we may need to install tools that are used during debian source building
that are not provided in the archive (such as rust crates, hello
cargo-vendor-filterer!!).

So instead of polluting this generic action with package-specific
requirements, let's just make it support running a script that can do
more advanced configurations in the docker instance that is used only
during the source-preparation phase.

This is required in order to be able to do a `cargo install --root=/usr
cargo-vendor-filterer` to build the source package of rust programs as
required by authd in ubuntu/authd#130

UDENG-2343
@3v1n0 3v1n0 force-pushed the debian-packaging branch 6 times, most recently from c5d815b to 57e16b0 Compare December 19, 2024 06:31
@3v1n0 3v1n0 requested a review from adombeck December 19, 2024 06:33
@3v1n0 3v1n0 marked this pull request as ready for review December 19, 2024 06:33
@3v1n0 3v1n0 requested a review from denisonbarbosa December 19, 2024 06:40
@3v1n0 3v1n0 dismissed didrocks’s stale review December 19, 2024 06:41

It was based on the old code, and comments have been addressed.

@3v1n0 3v1n0 force-pushed the debian-packaging branch 5 times, most recently from f2aecf5 to dee7b3a Compare December 19, 2024 15:41
Copy link
Member

@denisonbarbosa denisonbarbosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for handling this!

@3v1n0 3v1n0 force-pushed the debian-packaging branch from dee7b3a to c386833 Compare January 9, 2025 03:00
@3v1n0 3v1n0 requested a review from didrocks January 9, 2025 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants