test_runner: add bail out #56490

pmarchini · 2025-01-06T19:15:47Z

Catching up with the last attempt (#48919), this is another try at introducing the bailout feature.
I'm opening this PR as a draft to discuss the implementation and because refactoring may be needed if this approach is well-received by the community.

Note: In some tests, I had to enforce a concurrency=1 setting because testing the bailout feature across multiple files concurrently proved to be extremely flaky.

nodejs-github-bot · 2025-01-06T19:15:54Z

Review requested:

@nodejs/test_runner

atlowChemi · 2025-01-06T19:50:41Z

lib/internal/test_runner/runner.js

+      this.root.harness.testsProcesses.forEach((child) => {
+        child.kill();
+      });
+      return;


replace with an abort signal?

hey @atlowChemi, sure!

We already have an existing signal passed to the spawned process, but that is a signal at the Test level IIRC. perhaps we can add a new controller on root.harness, and spawn with a AbortSignal.any

cjihrig · 2025-01-06T19:59:57Z

In some tests, I had to enforce a concurrency=1 setting because testing the bailout feature across multiple files concurrently proved to be extremely flaky.

Can you explain more about what was flaky? I'm guessing you mean the tests were at different points of execution when they received the bail out signal. I think the best way to work around this is to use test fixtures that never finish.

pmarchini · 2025-01-06T20:08:53Z

Can you explain more about what was flaky? I'm guessing you mean the tests were at different points of execution when they received the bail out signal. I think the best way to work around this is to use test fixtures that never finish.

Hey @cjihrig, you're guessing right!

Your proposed solution sounds good.

I think we should add a test as follows:

First file test: A test that fails after a long timeout (maybe 5-10 seconds) to allow other file test processes to be spawned correctly.
Second file test: A test with an infinite loop.

WDYT?

cjihrig · 2025-01-06T20:06:25Z

lib/internal/test_runner/runner.js

+  .add(kCancelledByParent)
+  .add(kAborted)
+  .add(kTestTimeoutFailure)
+  .add(kTestBailedOut);


I know you just added to it here, but can we initialize the set without repeatedly calling add().

cjihrig · 2025-01-06T20:13:50Z

doc/api/test.md

+
+> Stability: 1 - Experimental
+
+The `--test-bail` flag provides a way to stop the test execution


Suggested change

The `--test-bail` flag provides a way to stop the test execution

The `--test-bail` flag provides a way to stop test execution

cjihrig · 2025-01-06T20:16:20Z

doc/api/test.md

+By enabling this flag, the test runner will exit the test suite early
+when it encounters the first failing test, preventing
+the execution of subsequent tests.
+Already running tests will be canceled, and no further tests will be started.


I'm not sure if we need to get into this small of a detail, but technically there will be a window of time where new tests may be started before being cancelled. Maybe we can say something like "The test runner will cancel all remaining tests."

cjihrig · 2025-01-06T20:16:40Z

doc/api/test.md

+when it encounters the first failing test, preventing
+the execution of subsequent tests.
+Already running tests will be canceled, and no further tests will be started.
+**Default:** `false`.


This default seems out of place.

cjihrig · 2025-01-06T20:18:04Z

doc/api/test.md

@@ -1483,6 +1512,11 @@ changes:
  does not have a name.
 * `options` {Object} Configuration options for the test. The following
  properties are supported:
+  * `bail` {boolean}


I don't think this should be an option to test(). Should it be part of run() instead?

Yes, my bad, the bail option it's part of run. I'll update the doc ASAP !

cjihrig · 2025-01-06T20:32:10Z

test/fixtures/test-runner/output/bail-spec.js

+});
+
+test('top level', (t) => {
+  t.test('forth', () => {});


Suggested change

t.test('forth', () => {});

t.test('fourth', () => {});

cjihrig · 2025-01-06T20:35:47Z

test/parallel/test-runner-run.mjs

-      fixtures.path('test-runner', 'never_ending_sync.js'),
-      fixtures.path('test-runner', 'never_ending_async.js'),
-    ] });
+    const stream = run({


This file seems to have a number of stylistic changes. Was that intentional?

Nope, I will remove unnecessary changes on the next commit!

cjihrig · 2025-01-06T20:38:28Z

test/parallel/test-runner-run.mjs

+    it('should only allow a boolean in options.bail', async () => {
+      [Symbol(), {}, [], () => { }, 0, 1, 0n, 1n, '', '1', Promise.resolve([])]
+        .forEach((bail) => assert.throws(() => run({ bail }), {
+          code: 'ERR_INVALID_ARG_TYPE'


Just out of curiosity, are we sure that these errors actually correspond to the use of bail? For example, it's possible that ERR_INVALID_ARG_TYPE or ERR_INVALID_ARG_VALUE are thrown by run() completely unrelated to the bail option.

I've written the test following a red-green cycle.
I'll take another look and I'll add a comment here 😁

Another option is to update the assertion to check the code and the message.

I agree, we should probably do the same for the other "validation" tests as well

cjihrig · 2025-01-06T20:39:07Z

test/parallel/test-runner-run.mjs

+
+  it('should handle the bail option', async () => {
+    const stream = run({
+      files: [join(testFixtures, 'bailout/sequential.test.mjs')],


Minor nit, but if we're going to use join() anyway:

Suggested change

files: [join(testFixtures, 'bailout/sequential.test.mjs')],

files: [join(testFixtures, 'bailout', 'sequential.test.mjs')],

cjihrig · 2025-01-06T20:40:59Z

test/parallel/test-runner-run.mjs

+
+  // TODO(pmarchini): Bailout is not supported in watch mode yet but it should be.
+  // We should enable this test once it is supported.
+  it.todo('should handle the bail option with watch mode');


Should this be documented?

cjihrig · 2025-01-06T20:52:06Z

I think we should add a test as follows:

First file test: A test that fails after a long timeout (maybe 5-10 seconds) to allow other file test processes to be spawned correctly.

Second file test: A test with an infinite loop.

There are all sorts of annoying edge cases to account for here, and it might be worth doing a survey of how the tap, mocha, and vitest runners handle bailing out when things are running in parallel. It's much more straightforward when only one thing is running. But, for example, if the very first test in the first process fails, do we bother spawning the other child processes at all? Or, in a bail out situation, how important is it to have an accurate summary at the end of the test run with correct counts for total tests, cancelled tests, etc.

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jan 6, 2025

test_runner: add bail out

55874e8

pmarchini force-pushed the test_runner/bail-out branch from 89f64db to 55874e8 Compare January 6, 2025 19:18

pmarchini requested review from cjihrig, MoLow, marco-ippolito and jakecastelli January 6, 2025 19:18

atlowChemi reviewed Jan 6, 2025

View reviewed changes

cjihrig reviewed Jan 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_runner: add bail out #56490

test_runner: add bail out #56490

pmarchini commented Jan 6, 2025

nodejs-github-bot commented Jan 6, 2025

atlowChemi Jan 6, 2025

pmarchini Jan 6, 2025

atlowChemi Jan 6, 2025

cjihrig commented Jan 6, 2025

pmarchini commented Jan 6, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

pmarchini Jan 7, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

pmarchini Jan 7, 2025

cjihrig Jan 6, 2025

pmarchini Jan 7, 2025

cjihrig Jan 8, 2025

pmarchini Jan 8, 2025

cjihrig Jan 6, 2025

cjihrig Jan 6, 2025

cjihrig commented Jan 6, 2025


		> Stability: 1 - Experimental

		The `--test-bail` flag provides a way to stop the test execution

	files: [join(testFixtures, 'bailout/sequential.test.mjs')],
	files: [join(testFixtures, 'bailout', 'sequential.test.mjs')],

test_runner: add bail out #56490

Are you sure you want to change the base?

test_runner: add bail out #56490

Conversation

pmarchini commented Jan 6, 2025

nodejs-github-bot commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjihrig commented Jan 6, 2025

pmarchini commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjihrig commented Jan 6, 2025