Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent test failures in Travis #49

Open
djones6 opened this issue Nov 30, 2018 · 3 comments
Open

Intermittent test failures in Travis #49

djones6 opened this issue Nov 30, 2018 · 3 comments

Comments

@djones6
Copy link
Contributor

djones6 commented Nov 30, 2018

The CircuitBreaker tests appear to have started intermittently failing in Travis. My suspicion is that this is test fragility rather than a real problem, but it ought to be investigated.

Example:
https://api.travis-ci.org/v3/job/461370473/log.txt

Test Case '-[CircuitBreakerTests.CircuitBreakerTests testStateCycle]' started.
/Users/travis/build/IBM-Swift/CircuitBreaker/Tests/CircuitBreakerTests/CircuitBreakerTests.swift:617: error: -[CircuitBreakerTests.CircuitBreakerTests testStateCycle] : XCTAssertEqual failed: ("closed") is not equal to ("open") - 
/Users/travis/build/IBM-Swift/CircuitBreaker/Tests/CircuitBreakerTests/CircuitBreakerTests.swift:620: error: -[CircuitBreakerTests.CircuitBreakerTests testStateCycle] : XCTAssertEqual failed: ("closed") is not equal to ("open") - 
/Users/travis/build/IBM-Swift/CircuitBreaker/Tests/CircuitBreakerTests/CircuitBreakerTests.swift:625: error: -[CircuitBreakerTests.CircuitBreakerTests testStateCycle] : XCTAssertEqual failed: ("closed") is not equal to ("halfopen") - 
Test Case '-[CircuitBreakerTests.CircuitBreakerTests testStateCycle]' failed (1.026 seconds).
@djones6
Copy link
Contributor Author

djones6 commented Nov 30, 2018

I've pushed a branch to trigger a Travis build so we can see some examples:
https://travis-ci.org/IBM-Swift/CircuitBreaker/builds/461718861

@djones6
Copy link
Contributor Author

djones6 commented Mar 26, 2019

Another recent intermittent failure: https://travis-ci.org/IBM-Swift/CircuitBreaker/jobs/511393203

Test Case 'CircuitBreakerTests.testBulkheadFullQueue' started at 2019-03-26 09:42:25.301
/home/travis/build/IBM-Swift/CircuitBreaker/Tests/CircuitBreakerTests/CircuitBreakerTests.swift:580: error: CircuitBreakerTests.testBulkheadFullQueue : XCTAssertEqual failed: ("2") is not equal to ("3") - 
Test Case 'CircuitBreakerTests.testBulkheadFullQueue' failed (2.006 seconds)

@djones6 djones6 mentioned this issue Apr 4, 2019
@Andrew-Lees11 Andrew-Lees11 self-assigned this Apr 15, 2019
@Andrew-Lees11 Andrew-Lees11 modified the milestones: 2019.08, 2019.09 Apr 23, 2019
@Andrew-Lees11 Andrew-Lees11 modified the milestones: 2019.09, 2019.10 May 8, 2019
@Andrew-Lees11 Andrew-Lees11 modified the milestones: 2019.10, 2019.11 May 22, 2019
@Andrew-Lees11 Andrew-Lees11 modified the milestones: 2019.11, 2019.12 Jun 5, 2019
@Andrew-Lees11 Andrew-Lees11 modified the milestones: 2019.12, 2019.13 Jun 19, 2019
@Andrew-Lees11
Copy link
Contributor

The circuit breaker tests are run relying on them being synchronously when they are async. This tries to be enforced using sleep but I think this doesn't always work causing the intermittent failures.
I attempted to fix this in this in pr #52 however even when using dispatch groups I was getting test failures.
I am unable to work in this further but will leave the PR up in case someone wants to look at in when they come back to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants