Non exhaustive testing gotchas

[evilhonda3030]

My project is getting bigger and exhaustive tests slows us down

The symptoms are:

1. Test are extremely fragile
2. Test are giant
3. Real expectations are hard to read

All in all, all that stuff Krzysztof Zabłocki uncovered once upon a time.

---

To mitigate I started experimenting with exhaustivity == .off and it turned out to be less intuitive than I expected

Here is the list of the things I'm concerned about:

A little information on how to wait/control effects a sent action causes

Use case

I want send and action, wait for its effects to finish and assert on the results aka I don't want to care about implementation details.

Issues

1. assert docs say we can do the following

await store.send(\.child.closeButtonTapped)
await store.finish()
await store.skipReceivedActions()
store.assert {
  $0.child = nil
}

While finish() docs say that post interactions with TestStore cause undefined behaviour.

2. Sometimes I can't call finish since I want to interact with TestStore more and no documentation says how I can proceed.

• Awaiting TestStoreTask does not apply state changes. Is it a limitation of having receive API?
• Neither TestStore.finish() does. I don't know if it's actually correct since finish() declares that further interactions are impossible.
• skipInFlightEffects() does not support timeout, I assume megaYield might be not sufficient.

Summary

1. Can we add a solution below as inline docs for TestStoreTask.finish? Or add a helper method to the library?

await store.send(...).finish()
await store.skipReceivedActions()
await store.skipInFlightEffects()

func sendAndWaitForEffects(.)

2. .finish() is not required if there are few effects thanks to megaYield. But I have to admit that relying on implementation details is dangerous. I was able to implement a test (rather insane one) to break megaYield. Would be great to communicate this through the docs.

await store.send(...) // without finish
await store.skipReceivedActions()

3. Should skipInFlightEffects() support timeout?

4. Does this have undefined behaviour?

await store.send(...)
await store.finish()
await store.receive(...)

---

Assert action is NOT received

Use case

Filter out unnecessary events of AsyncStream in .run to avoid performance pitfalls.

Issues

1. I cannot use XCTExpectFailure due to lack of Concurrency support.

XCTExpectFailure { 
   await store.receive(...)
}

Summary

1. I would love TCA has APIs to do the following

await store.send(...).finish() // causes subscription to the stream
XCTAssertFalse(store.hasRecieved(....)) 

---

Fragile tests due to unimplemented dependencies

Use case

I don't want my test to break due to use of a new dependency in Feature unless expectation fails

Summary

1. Should we not use testValue at all and make testValue equal to previewValue when defining dependency? The downside is bad support of exhaustive testing when needed.
2. Should we do store.dependencies = .preview at the beginning of the test?
3. Are there other ways to neatly work it out?

---

I'm attaching a simple project which implement some tests which fail.

NonExhaustiveTestingGotchas.zip

[evilhonda3030]

I'll close this discussion and open 3 topics-focused ones. I'll also reformulate my proposals for more productive discussion.