Allow veto of certain combinations

[AArnott]

PairwiseDataAttribute and CombinatorialDataAttribute should accept a parameter that allows the test to veto certain known invalid parameter combinations. Thereby suppressing generation of test cases that don't make sense.

Ideally in the pairwise case, such a suppression will feed back into the pairwise algorithm so it can try selecting other combinations that still satisfy pairwise testing requirements, wherever possible.

[ilyabreev]

+1 for this feature. It seems not really hard to implement it at least for CombinatorialDataAttribute.

[alrz]

+1

Currently we have to do switch (param1, param2) { case (invalid, combinations): .. return; } which doesn't prevent redundant test runs.

[AArnott]

Can someone make a proposal that includes the new API surface area, example uses and describe the behavior?

[alrz]

This is the simplest I could come up with

[Theory, CombinatorialData]
[CombinatiorialExclude(false, false, false)]
// maybe a few more
public void CheckValidAge(bool p1, bool p2, bool p3)

to exclude (false, false, false) test case.

Something similar should probably work for pairwise, but I don't have a good use case for it right now.

[AArnott]

That's a good start. What about when the args can't be expressed as constants? Something like MemberData provides. Or maybe that already doesn't work for Combinatorial, I don't recall.

[alrz]

What about when the args can't be expressed as constants? Something like MemberData provides. Or maybe that already doesn't work for Combinatorial,

Even if it does work, that may need a special equality check as well so it's probably out of scope. I think an attribute like that will cover 90% of scenarios.

[AArnott]

#1 actually tracks a MemberData-like attribute and it looks like we may have a volunteer for contributing it soon.

[AndrewKitu]

Maybe to add 2 different attributes?

CombinatiorialExclude for constants and CombinatiorialExcludeMemberData for complex data.

[Theory, CombinatorialData]
[CombinatiorialExclude(1, 1, 1)]
public void CheckFailIfAnyParamIsNonpositive(
    [CombinatorialValues(-1, 0, 1)] int a, 
    [CombinatorialValues(-1, 0, 1)] int b, 
    [CombinatorialValues(-1, 0, 1)] int c)

[Theory, CombinatorialData]
[CombinatiorialExcludeMemberData(nameof(GetUntestableCombinations))]
public void CheckComplexParameters(
    [CombinatorialMemberDataValues(nameof(GetComplexData1))] Complex1 complex1, 
    [CombinatorialMemberDataValues(nameof(GetComplexData2))] Complex2 complex2)

[AArnott]

Sure, that could work.

[alrz]

26696c1 doesn't seem to be related?

[AArnott]

Agreed. That's a mistake from merging commits from another repo. Thanks for catching, @alrz.

[relfman-cmg]

+1

Personally I would need something like [CombinatiorialExcludeMemberData(nameof(GetUntestableCombinations))]