Use npm instead of yarn to build Theia #14481

tsmaeder · 2024-11-19T13:23:49Z

What it does

This PR rewrites the Theia build process to use npm in stead of yarn. This required a couple of changes:

Contrary to yarn, npm does not invoke the prepare scripts of workspace packages in the order of their dependencies (npm/cli#3034). So instead of relying on yarn install to invoke the necessary build scripts, we have to invoke the build process explicitly after the npm install. The build process is built on this logic:

Each package has a script called "build". The goal of that target is to fully build all artifacts this repo produces. For regular Theia packages, this is just incrementally compiling the typescript sources of the package an it's dependencies, but for applications, for example, it means creating the application bundle. So calling "build" in an application package should be all that's needed in regular development
To build the repo initially after check-out, there are a couple of additinal steps necesary:
- compile build tools (re-exports package)
- generate re-exports where applicable
- compile all packages
- build alll application packages
  This can be achieved by invoking npm run build in the repo root.

Fixes #13948

Contributed on behalf of STMicroelectronics

There are a couple of issues in npm that currently require manual edits of package-lock.json

bug: overrides property only honored when running install the first time npm/cli#5850
Also: npm is not hoisting packages that are only used in a single workspace. This breaks stuff like NativeWebpackPlugin.

How to test

Make sure we can build Theia as described above and the apps work fine.

Follow-ups

Make NativeWebpackPlugin resilient to Hoisting #14707

Attribution

Contributed on behalf of STMicroelectronics

Review checklist

As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

As a reviewer, I agree to behave in accordance with the review guidelines

dankeboy36 · 2024-11-24T17:32:41Z

Fixes #13948

Possible related: #775

tsmaeder · 2024-11-25T15:44:03Z

@msujew @sdirix the PR checks now pass and it's ready to be reviewed minus doc.

sdirix

So far it works very well.

I used npm ci to install. I tested all applications and was able to build and start all of them. I also tested watching for the browser application. Tests also completed successfully.

Linting is currently broken but this is already mentioned in the current PR description.

@theia/request: > @theia/[email protected] lint
@theia/request: > theiaext lint
@theia/request: $ eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"
@theia/request: /bin/sh: 1: node: not found
@theia/request: Oops! Something went wrong! :(
@theia/request: ESLint: 8.57.1
@theia/request: SyntaxError: Failed to load plugin '@theia' declared in '.eslintrc.js » ../../configs/build.eslintrc.json » ./base.eslintrc.json': Unexpected end of JSON input
@theia/request:     at JSON.parse (<anonymous>)
@theia/request:     at PackageReExports.FromPackageSync (/home/user/git/theia/dev-packages/private-re-exports/lib/package-re-exports.js:127:47)
@theia/request:     at Object.<anonymous> (/home/user/git/theia/dev-packages/private-eslint-plugin/rules/shared-dependencies.js:24:40)

sdirix · 2024-11-27T20:14:45Z

.editorconfig

@@ -1,7 +1,6 @@
 root = true

 [*]
-insert_final_newline = true


Can we keep this? Why remove it?

Because we now have the preference "files.insertFinalNewline" which acomplishes the same thing.

.github/workflows/ci-cd.yml

dev-packages/application-manager/src/generator/webpack-generator.ts

dev-packages/private-ext-scripts/package.json

package.json

dev-packages/private-ext-scripts/package.json

package.json

msujew

Works pretty well, but there are still a few calls to yarn littered throughout the build scripts:

theia/scripts/run-reverse-topo.js

Line 34 in 37d8330

const YARN_WORKSPACES = JSON.parse(cp.execSync('yarn --silent workspaces info').toString());
theia/scripts/translation-update.js

Line 42 in 37d8330

const childProcess = cp.spawnSync('yarn', [

There might be more. See also below.

packages/electron/package.json

scripts/compile-references.js

tsmaeder · 2024-12-19T14:09:00Z

@JonasHelming in response to your question elsewhere, the documentation on https://theia-ide.org/ mostly uses the yeoman generator output and the theia-ide repository as examples. While the output of the generator and theia-ide still use yarn, it makes no sense to update the documentation.

tsmaeder · 2025-01-06T11:31:44Z

@msujew @sdirix could I get some love for this one, please? If we merge it, we should merge it early in the release cycle.

msujew

Looks good to me know. Everything seems to work as expected without relying on yarn.

sdirix · 2025-01-07T11:18:46Z

Linting fails for me with

Node 20.18.1
npm 10.8.2
Ubuntu 22.04

It errors in all packages with errors like this:

> @theia/[email protected] lint
> theiaext lint

$ eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"
/bin/sh: 1: node: not found

Oops! Something went wrong! :(

ESLint: 8.57.1

SyntaxError: Failed to load plugin '@theia' declared in '.eslintrc.js » ../../configs/build.eslintrc.json » ./base.eslintrc.json': Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at PackageReExports.FromPackageSync (/home/user/Git/theia/dev-packages/private-re-exports/lib/package-re-exports.js:127:47)
    at Object.<anonymous> (/home/user/Git/theia/dev-packages/private-eslint-plugin/rules/shared-dependencies.js:24:40)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/home/user/Git/theia/dev-packages/private-eslint-plugin/index.js:24:28)
Exit with failure status (2): eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"
npm error Lifecycle script `lint` failed with error:
npm error code 2
npm error path /home/user/Git/theia/packages/core
npm error workspace @theia/[email protected]
npm error location /home/user/Git/theia/packages/core
npm error command failed
npm error command sh -c theiaext lint

Linting works for me in the current yarn based mono repo.

msujew · 2025-01-07T11:47:52Z

@sdirix I had the same issue, it's related to some cached info. You need to run git clean -xfd once before running npm i.

sdirix · 2025-01-07T12:14:30Z

@sdirix I had the same issue, it's related to some cached info. You need to run git clean -xfd once before running npm i.

Sadly I already tried that, the issue persists. Do you know how exactly the @theia import in the eslint config resolves? It seems that this step fails for me.

tsmaeder · 2025-01-07T12:19:34Z

@sdirix what's your invocation sequence for lint?

sdirix · 2025-01-07T13:05:21Z

I tried

npm ci
npm run lint

I also tried

npm ci
npm run compile
npm run lint

npm install
npm run lint

every time preceded by git clean -dfx

I also tried

npm ci
cd packages/core
npx eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"

I also tried changing @theia to @theia/eslint-plugin in the eslint config and I also tried removing @theia completely in the eslint config. These are then failing with different errors.

Fixes eclipse-theia#13948 Contributed on behalf of STMicroelectronics Signed-off-by: Thomas Mäder <[email protected]>

Signed-off-by: Thomas Mäder <[email protected]>

See eclipse-dash/dash-licenses#415 Signed-off-by: Thomas Mäder <[email protected]>

Signed-off-by: Thomas Mäder <[email protected]>

sdirix

Works for me now! Thanks for the fix.

Once we merge we should do a prerelease so we can do some real world testing in our downstream repositories (which are still yarn based) to verify that they continue working.

Signed-off-by: Thomas Mäder <[email protected]>

eclipse-theia/theia#14481

eclipse-theia/theia#14481 Contributed on behalf of STMicroelectronics

tsmaeder requested review from msujew and sdirix November 19, 2024 13:23

tsmaeder force-pushed the npm_experiment branch 2 times, most recently from 70c2767 to 82b7da0 Compare November 24, 2024 12:07

dankeboy36 mentioned this pull request Nov 24, 2024

consider moving from yarn to npm arduino/vscode-arduino-tools#20

Open

sdirix reviewed Nov 27, 2024

View reviewed changes

msujew reviewed Dec 3, 2024

View reviewed changes

packages/electron/package.json Show resolved Hide resolved

scripts/compile-references.js Outdated Show resolved Hide resolved

tsmaeder force-pushed the npm_experiment branch from c75a1a9 to 937f18b Compare December 17, 2024 17:03

tsmaeder marked this pull request as ready for review December 18, 2024 15:24

tsmaeder requested a review from sdirix December 18, 2024 15:24

tsmaeder requested a review from msujew January 3, 2025 10:20

tsmaeder force-pushed the npm_experiment branch from 0e3f7af to b3af23a Compare January 3, 2025 14:14

msujew approved these changes Jan 6, 2025

View reviewed changes

tsmaeder added 8 commits January 7, 2025 15:35

Use npm instead of yarn to build Theia

3ba64e5

Fixes eclipse-theia#13948 Contributed on behalf of STMicroelectronics Signed-off-by: Thomas Mäder <[email protected]>

Post-rebase fixes

6c38731

Signed-off-by: Thomas Mäder <[email protected]>

Update dependency check baseline to include plugins

673c6d1

Signed-off-by: Thomas Mäder <[email protected]>

Address review comments

6786548

Signed-off-by: Thomas Mäder <[email protected]>

Add final newline

5c8fd2c

Signed-off-by: Thomas Mäder <[email protected]>

Getting rid of references to "yarn"

1f6192d

Signed-off-by: Thomas Mäder <[email protected]>

Remove workspace plugins from dependency-baseline

8bfbdd2

See eclipse-dash/dash-licenses#415 Signed-off-by: Thomas Mäder <[email protected]>

Update package-lock.json after rebase

51115e8

Signed-off-by: Thomas Mäder <[email protected]>

Use argv[0] to launch sub-scripts

e0fd986

Signed-off-by: Thomas Mäder <[email protected]>

sdirix approved these changes Jan 7, 2025

View reviewed changes

Update package-lock.json post-rebase

9b81e92

Signed-off-by: Thomas Mäder <[email protected]>

tsmaeder force-pushed the npm_experiment branch from ff7c76f to 9b81e92 Compare January 7, 2025 15:08

msujew mentioned this pull request Jan 8, 2025

Improve bundling for hoisted dependencies #14708

Open

1 task

tsmaeder merged commit 0036769 into eclipse-theia:master Jan 8, 2025
12 of 13 checks passed

github-actions bot added this to the 1.58.0 milestone Jan 8, 2025

planger added a commit to eclipsesource/theia-e2e-test-suite that referenced this pull request Jan 9, 2025

fix: adapt for npm switch in theia

1ddaf1c

eclipse-theia/theia#14481

planger mentioned this pull request Jan 9, 2025

fix: adapt for npm switch in theia eclipse-theia/theia-e2e-test-suite#44

Draft

planger added a commit to eclipsesource/theia-e2e-test-suite that referenced this pull request Jan 9, 2025

fix: adapt for npm switch in theia

1b50ee7

eclipse-theia/theia#14481 Contributed on behalf of STMicroelectronics

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use npm instead of yarn to build Theia #14481

Use npm instead of yarn to build Theia #14481

tsmaeder commented Nov 19, 2024 •

edited

Loading

dankeboy36 commented Nov 24, 2024

tsmaeder commented Nov 25, 2024

sdirix left a comment •

edited

Loading

sdirix Nov 27, 2024

tsmaeder Dec 16, 2024

msujew left a comment

tsmaeder commented Dec 19, 2024

tsmaeder commented Jan 6, 2025

msujew left a comment

sdirix commented Jan 7, 2025

msujew commented Jan 7, 2025

sdirix commented Jan 7, 2025

tsmaeder commented Jan 7, 2025

sdirix commented Jan 7, 2025

sdirix left a comment

Use npm instead of yarn to build Theia #14481

Use npm instead of yarn to build Theia #14481

Conversation

tsmaeder commented Nov 19, 2024 • edited Loading

What it does

How to test

Follow-ups

Attribution

Review checklist

Reminder for reviewers

dankeboy36 commented Nov 24, 2024

tsmaeder commented Nov 25, 2024

sdirix left a comment • edited Loading

Choose a reason for hiding this comment

sdirix Nov 27, 2024

Choose a reason for hiding this comment

tsmaeder Dec 16, 2024

Choose a reason for hiding this comment

msujew left a comment

Choose a reason for hiding this comment

tsmaeder commented Dec 19, 2024

tsmaeder commented Jan 6, 2025

msujew left a comment

Choose a reason for hiding this comment

sdirix commented Jan 7, 2025

msujew commented Jan 7, 2025

sdirix commented Jan 7, 2025

tsmaeder commented Jan 7, 2025

sdirix commented Jan 7, 2025

sdirix left a comment

Choose a reason for hiding this comment

tsmaeder commented Nov 19, 2024 •

edited

Loading

sdirix left a comment •

edited

Loading