Overriding Verdict of TestPlan

1

Hi everyone,

I developed a mixin for repeating a test step until specified verdict is obtained. If that mixin is added to the test step, test step is repeated many times until “Pass” verdict is obtained. Mixin works fine but verdict of the test plan is “Fail” due to the failed repeats.

Is there a way of overriding or re-evaluating the verdict of the test plan after all the test steps are executed? Any help is appreciated.

2

Hi @btyilmaz, you should probably set the verdict of the step to "Not Set"when you retry a step. Otherwise the parent step will gather all the child step runs and give the most severe verdict.

e.g:

public void OnPostRun(TestStepPostRunEventArgs step)
        {
      // ...
      step.Verdict = Verdict.NotSet;
     }
1 Like
3

Parent step’s verdict is set as I expected but test plan’s verdict is still not updated according to the last execution of the test steps. I am posting screenshots of test plan and log panel:

image

image
image732×140 3.89 KB

I want this test plan run’s verdict to be “Pass” however, result listener get the test plan verdict as “Fail”

4

I see. Are you repeating the sequence step or the RepeatMixinTest step?

5

I am repeating the Sequence step

6

The problem is that the test plan is calculating the verdict based on all its child step runs. It will select the most severe verdict in the list of runs.

You can get around this by either:

  • Overriding the TestStepRun verdict for failed steps. (I recommend this, did you try it?)
  • Creating a parent step which does something else with the verdicts of the child test steps.
7

I just tried overriding TestStepRun verdict’s but I can’t access Verdict property of TestStepRun class because its set accessor is internal. I tried to set it inside the OnPostRun method of the repeat mixin.

8

@btyilmaz, in this case, I did not mean overriding through inheritance. I guess I really meant “overwriting”. You should be able set the verdict directly on the test step in the mixin event args.

9

It didn’t work either. I looked over the OpenTap’s test plan execution and TestPlan’s verdict is calculated at the end of the test plan execution using the test step runs. Since test step runs’ verdicts are fail, they always make test plan’s verdict fail.

Now, I am not sure how to get around this since test step runs’ verdict property are not accessible.

10

I just verified that this works. Do you remember to use the EmbedProperties attribute?

   [Test]
        public void SetVerdictMixinTest()
        {
            var verdict = new VerdictStep()
            {
                VerdictOutput = Verdict.Fail
            };

           // this is the same as adding the mixin with a UI.
            MixinFactory.LoadMixin(verdict, new SetVerdictMixinBuilder());

            var plan = new TestPlan();
            plan.ChildTestSteps.Add(verdict);
            var run = plan.Execute();
            Assert.AreEqual(Verdict.Pass, run.Verdict);
        }

    }

    public class SetVerdictMixinBuilder : IMixinBuilder
    {
        public void Initialize(ITypeData targetType)
        {
            
        }

        public MixinMemberData ToDynamicMember(ITypeData targetType)
        {
            return new MixinMemberData(this, () => new SetVerdictMixin())
            {
                TypeDescriptor = TypeData.FromType(typeof(SetVerdictMixin)),
                Name = "VerdictSetter",
                Readable = true,
                Writable = false,
                Attributes = new []{new EmbedPropertiesAttribute()}
            };
        }
  
    public class SetVerdictMixin : IMixin, ITestStepPostRunMixin
    {
        public Verdict Verdict { get; set; } = Verdict.Pass;
        public void OnPostRun(TestStepPostRunEventArgs eventArgs)
        {
            eventArgs.TestStep.Verdict = Verdict;
        }
    }
11

You are right, it works. I forgot that I added another mixin to set the verdict of a parent step. Following mixin is the reason for not updating the verdict.

[Display("Set Last Verdict", "Mixin for upgrading verdict according to the last verdicts of the child steps..")]
public class SetLastVerdictMixin : ITestStepPostRunMixin
{
    [Display("Enabled")] public bool IsEnabled { get; set; } = true;
    public void OnPostRun(TestStepPostRunEventArgs eventArgs)
    {
        if(!IsEnabled)
            return;

        var step = eventArgs.TestStep;
        var childStepsVerdicts = step.ChildTestSteps.Select(x => x.Verdict);
        step.Verdict = Verdict.Pass;

        foreach (var childStepsVerdict in childStepsVerdicts)
        {
            UpgradeVerdict(step, childStepsVerdict);
        }
    }

    private static void UpgradeVerdict(ITestStep step, Verdict newVerdict)
    {
        if (step.Verdict < newVerdict)
            step.Verdict = newVerdict;
    }
}
1 Like