I believe I understand how the ‘Break Conditions’ work for test steps but I don’t think they will accomplish what I need.
What I am trying to do is cause a specific test that is nested below several layers of parent steps to exit a test on failure without causing other sibling steps to exit on failure.
I am trying to do this with existing steps using only the ‘Break Conditions’ based on Verdict in order to be a generic pattern I can use.
With a custom test step, I understand I could just throw an exception to accomplish this.
A simple example:
In the above, I want the test to end if DebugStep (2) fails, but not if DebugStep (3) fails.
I may be missing something, but I don’t see any combination of ‘Break Conditions’ that will accomplish this. Am I correct?
Thanks,
Wittrock
P.S. : I am considering if a Mixin could be used to accomplish this generically. Unfortunately, I am using EditorX 1.5.0 for the present and it does not have Mixin support for editing though I can hand modify the testplan.
This is a simple mixin to accomplish the above.
public class EndTestMixin : ITestStepPostRunMixin
{
[Display("End Test Condition", "Conditions on which to end the test")]
public BreakCondition BreakOn {get; set;}
public void OnPostRun (TestStepPostRunEventArgs eventArgs)
{
Verdict verdict = eventArgs.TestStep.Verdict;
if (((Verdict.Fail == verdict) && BreakOn.HasFlag(BreakCondition.BreakOnFail)) ||
((Verdict.Error == verdict) && BreakOn.HasFlag(BreakCondition.BreakOnError)) ||
((Verdict.Inconclusive == verdict) && BreakOn.HasFlag(BreakCondition.BreakOnInconclusive)) ||
((Verdict.Pass == verdict) && BreakOn.HasFlag(BreakCondition.BreakOnPass))
)
{
eventArgs.TestStep.PlanRun.MainThread.Abort();
}
}
}
[Display("Test End", "Ends test based on verdict")]
[MixinBuilder(typeof(ITestStepParent))]
public class EndTestMixinBuilder : IMixinBuilder
{
public string MemberName{get;set;} = "End Test";
public void Initialize(ITypeData type)
{
}
public MixinMemberData ToDynamicMember(ITypeData targetType)
{
return new MixinMemberData(this, () => new EndTestMixin()) {
Name = nameof(EndTestMixinBuilder) + "." + MemberName,
TypeDescriptor = TypeData.FromType(typeof (EndTestMixin)),
Attributes = new object[]{new DisplayAttribute(MemberName), new EmbedPropertiesAttribute()},
DeclaringType = TypeData.FromType(typeof(ITestStep))
};
}
}
Currently I call TestStep.PlanRun.MainThread.Abort() to exit the test.
Is there an convenient/alternative way to end the test that does not cause the final Verdict to be an Error or Abort?
In other words, I want to end the test on certain test Failure, but I don’t necessarily want to report this as an Abort or Error since it may be a perfectly expected end of the test with a Fail status expected.
Hi @wittrock,
Instead of calling MainThread.Abort, or throwing an exception which is the way i recommend stopping the test plan.
You can set the verdict to fail and set the break condition to “Break on Fail”
Here for example, I made a test plan with dialog steps inside a sequence. The dialog step sets the verdict to fail when I click cancel, and it has break condition set to Break on Fail". Notice how the last dialog step never executed:
Thanks @rolf_madsen ,
The problem I have with using the break condition seems to be that when I have a nested hierarchy, I cannot have a mix of steps that I want to end the test on a fail condition and those I do not want to exist the test on fail.
I tried to show that in my original post but it may not be clear.
Using your example, the problem is, in order to exit the test on failure of Dialog (1), I would also need to have its parent (Sequence) set to break on failure. If I do this, any sibling of Dialog (1) that fails will also cause the test to end when the Sequence break condition is evaluated.
I hope that makes sense.
Thanks,
Wittrock
Ok, I think that might be because you set the break condition on the sequence step. You can also just set it on the test steps that you want to break. In this case, I set break conditions to Break on Error on the first sequence step and Break on Fail on the next:
If you set the break condition on the parent step it has the same behavior because the child steps generally inheirt the parent steps break condition, but you can override that behavior, which is what the ‘checkbox’ does.
I still don’t think this accomplishes what I am after.
It may help to go back to my original example.
So Imagine I only want one step in this test plan to exit the test if it fails (i.e. not run any other steps if it fails). In this example, lets say that’s the Debug (2) step.
If i only set the Break on Fail condition for Debug (2), when it fails it will still run DebugStep (4) because the Break is only breaking for the current level in the hierarchy.
This is an example of running with the Break on Fail set on DebugSetp (2). As you see, DebugStep (4) still runs.
If I also set the Break on Error condition for the Sequence step, when DebugStep (2) fails, it will now exit the test as I want, but the side effect is that now a failure of DebugStep (3) will also exit the test which is what I do not want.
This is an example of running with Break on Fail set on DebugStep (2) and the Sequence, and DebugStep (3) explicitly set to Not break on Fail ( not inheriting the sequence break condition). As you can see, it exits the test as I need.
This is the same setup, only now with a failure on DebugStep (3). As you can see, it has existed the test by not running DebugStep (4) and this is what I do not want.
To state it simply, I don’t see a way to end a test using a break condition that is independent of its location in the test step hierarchy.
If there is no Heirarchy (all steps at the top level or flat), then it is possible.
Again, I hope that is clear.
Thanks
Wittrock
Ok, yes, I get it now and I agree, that is not possible if you insist on that specific test plan layout. What you ask for don’t really fit the current model.
Implementing it as a custom test step plugin could also get a bit tricky. If you implement your own “CustomSequenceStep”, you could set SuggestedNextStep on the resulting test step run from the CustomSequenceStep, but it would not be able to end the test plan execution completely - only the current level, so for example the topmost sequence. Is something like that a possibility for you?
Thanks @rolf_madsen ,
Yes, the layout of test steps is just a contrived example.
The actual test plans I have are rather complicated and I end up using the Hierarchy for a number of things like organizing steps into sequences that are optionally disabled etc. so sometimes it is difficult to isolate steps that I want to end a test plan run or to define up front where they will be in the hierarchy.
From the standpoint of having the ability to exit a test using existing steps, the Mixin I show above works well. I think the only minor annoyance is that it ends the test with an Abort status. That’s not a deal breaker, but something I was hoping to avoid.
Thanks for your help.
Wittrock
