-
Notifications
You must be signed in to change notification settings - Fork 148
EventWatcher: reject waitingFor promise during test cleanup. #225
base: master
Are you sure you want to change the base?
Conversation
EventWatcher installs a test cleanup function that removes its event listeners. For this reason, if the EventWatcher's wait_for created a promise that is still pending when the test ends, the promise will never be resolved or rejected. This omission turns into a problem when the EventWatcher's wait_for promise ends up in the promise chain used by promise_test. If an assertion fails in a step() callback, the test's done() method is called, which cleans up the EventWatcher's listener. So, a failure in a single promise_test can block the whole promise chain and prevent all subsequent tests from running. This commit fixes the issue described above by having any pending EventWatcher promise get rejected when the event listeners get removed.
So does this fix #187? |
I think that my patch guarantees that EventWatcher's promise will resolve/reject, as long as the event fires. I suspect that it will solve the situation that #187 ran into, but I can't be sure without seeing a code snippet. Specifically, I haven't thought about the impact on assertion failures in promises (where the promise would get rejected) or in errors that aren't wrapped in a Test.step() call. Could we ask #187 for more input? |
I don't think you should worry about errors that aren't wrapped in a Test.step() call (and that don't ultimately result in the promise passed to promise_test to reject). But it would be nice if the solution would work for any assertion failure wrapped in Test.step(). Maybe Test can have a promise that is resolved/rejected when the test passes/fails, and promise_test can then race that promise with the promise passed to promise_test? That way you'll catch any situation where an assertion fails outside of the promise chain in a promise_test. Haven't thought about any unintended consequences too much, but it seems that running more tests, even if later results might be unreliable would still be preferred over just not running more tests at all. |
My fix will definitely work for errors wrapped in Test.step(), because the I think that my PR is orthogonal to the fix that you're proposing. I think that EventWatcher's promises should resolve/reject as predictably as possible, to facilitate debugging. I can work on a PR with the fix that you're proposing :) |
I created PR #226 with the fix that you proposed above. |
But with that fix, would purpose does this PR still serve? The only observable differences I can think of would be in cases where EventWatcher isn't used in a promise_test. But in that case I'm not convinced that rejecting the promise in all cases when the test is done is the right thing to do. |
The API for From a different angle: WDYT? |
I don't think it actually does, and I agree that that seems like a weird bug. wait_for never rejects. Instead when it receives an event that doesn't match the expected type it'll have an assertion failure (and without either of the fixes than no other promise tests would get run). I definitely think it makes sense for wait_for to be fixed to reject when one of the EventWatcher assertions failed. I'm still not convinced that it makes sense to reject when any other (possibly unrelated) assertion fails. I almost wonder if the whole stop_watching shouldn't be there, and instead EventWatcher should be a pure Promise based thing that either resolves or rejects, but doesn't actually effect a Test instance. But that might lead to people forgetting to check for promise rejection when used outside of promise_test. So probably a bad idea too. And for what it's worth, the "stop all test code from running when the first assertion failed" logic that is build in to Test.step would already have the effect of no more assertion failures and/or promise resolving after done() is called (step becomes a no-op after the test finished). |
I have hit an issue with eventWatcher, it doesn't seem to detect taps sometimes, but sticking a timeout in the test after wait_for call of about 5 seconds allows the eventWatcher to settle and detect all taps. Also it does not reject when the event being looked for is not thrown, this may be by design, but a reject with message is preferred. |
EventWatcher installs a test cleanup function that removes its event listeners. For this reason, if the EventWatcher's wait_for created a promise that is still pending when the test ends, the promise will never be resolved or rejected.
This omission turns into a problem when the EventWatcher's wait_for promise ends up in the promise chain used by promise_test. If an assertion fails in a step() callback, the test's done() method is called, which cleans up the EventWatcher's listener. So, a failure in a single promise_test can block the whole promise chain and prevent all subsequent tests from running.
This commit fixes the issue described above by having any pending EventWatcher promise get rejected when the event listeners get removed.
Test case below. If
assert_true(false
is replaced withassert_true(true
, both tests pass cleanly. With this PR, the first test fails and the second test passes. Without the PR, the first test times out and the second test never gets to run.This change is