Seems multiple layers of nested reference plans will prevent Editor to open the TestPlan. Anyone knows a workaround about this issue as it prevent me to create some modulised test plans? Thank you very much.
An example:
1, Take a test plan (level_1) with 5 log output, and parameterize each "Log Message " to the TestPlan
2, Load level_1 in a reference plan (level_2), and parameterize all “Log Message” as well.
3. Load level_2 in a reference plan (level_3) , and parameterize all “Log Message”, and repeat this structure until level_8
The loading of those test plans works good. Until level_7, Editor starts to struggle to open the test plan. And open level_8 takes more than 30 minutes on my work station.
The level_8 test plan self is actually quit simple if opened as text.
<?xml version="1.0" encoding="utf-8"?>
<TestPlan type="OpenTap.TestPlan" Locked="false">
<Steps>
<TestStep type="ref@OpenTap.Plugins.BasicSteps.TestPlanReference" Version="9.16.4+654f0b6b" Id="11bbd9e6-7d71-4f08-b9ce-cfc96e126f71">
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_1 Parameter="Parameters \ Log Message 1"></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_1>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_2 Parameter="Parameters \ Log Message 2"></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_2>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_3 Parameter="Parameters \ Log Message 3"></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_3>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_4 Parameter="Parameters \ Log Message 4"></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_4>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_5 Parameter="Parameters \ Log Message 5"></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_5>
<Filepath>level_7.TapPlan</Filepath>
<StepMapping />
<Enabled>true</Enabled>
<Name>Test Plan Reference</Name>
<BreakConditions>Inherit</BreakConditions>
</TestStep>
</Steps>
<BreakConditions>Inherit</BreakConditions>
<OpenTap.Description />
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_1></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_1>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_2></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_2>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_3></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_3>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_4></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_4>
<Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_5></Parameters_x0020__x005C__x0020_Log_x0020_Message_x0020_5>
<Package.Dependencies>
<Package Name="OpenTAP" Version="9.16.4+654f0b6b" />
</Package.Dependencies>
</TestPlan>
Hi Mark,
I was able to reproduce your issue.
We do seem to reach a usability limit (where the editor freezes and behaves very sluggishly) when you nest test plan references that deep.
Can you elaborate more on the modularizing aspect of what you are trying to do with so many levels of nested test plan references?
@rolf_madsen Maybe you can also chime in on whether there are any concrete usability limits to what Mark is trying to do here and if there is any alternate way?
1 Like
We previously had performance issues and this was actually improved a lot in 9.16, however I think the comfortable working level may still be around 5 levels of nested test plans. I think we’d have to start cut off features for that to work effectively.
I’d recommend creating a performance issue on gitlab.com/opentap/opentap with the test plans attached.
Hi @kaushik.santhanam and @rolf_madsen,
Sorry for this too much exaggerated case, in my project, I don’t use that much level of nesting either, I just want to show the issue in a simple way. It still happens when I increase the number of parameters in level_1, then I will have the problem to open the Test Plan Setting of level_5.
To demonstrate it I increase the parameters in level_1 to 15:
Even with just 5 levels, Editor still struggles to open the Test Plan Setting of test plan.
level_1.TapPlan (7.8 KB)
level_2.TapPlan (4.8 KB)
level_3.TapPlan (4.8 KB)
level_4.TapPlan (4.8 KB)
level_5.TapPlan (4.8 KB)
I guess that is somehow expected, if you reduce the level but then add enough parameters you hit the same bottleneck.
As I mentioned, add an issue on gitlab. This forum is not really the right place to track performance issues.
No problem, I’ll do that.
1 Like
@MarkLong let us know if you have any issues with Gitlab. A direct link to creating a new issue is here:
https://gitlab.com/OpenTAP/opentap/-/issues/new?issue[milestone_id]=
It sounds like you were testing some more advanced cases, but does the existing behavior meet your needs or would you need either the deeper nester or the additional parameters?
Hi @brennen_direnzo and @rolf_madsen ,
Thanks for giving the link, I have opened a issue there. Parameterization on reference plans freeze the Editor (#837) · Issues · OpenTAP / OpenTAP · GitLab
We are actually a customer of Keysight ev emulator system. And I find OpenTap project is also very interesting. I’m doing a POC project try to use OpenTap with python plugin to run our tests. So far so good and it has a nice graphic interface. Just those performance issue prevent us fully migrating to OpenTap eco system.
2 Likes
Ah really cool.! We work with that team closely obviously. Can you explain a bit of what it is you are trying to do? We may be able to suggest some alternative approaches at least until we are able to look into the performance.
Thanks for asking! We have some rather complex and big test suites. With lots of steps, the test runs sometimes for couple of days.
The reason we have so many parameters is our DUTs has different variants. So each test suite should support them with different parameters passing down to the steps.
Here is a simple illustration of our test suites, Please let me know if anything is not clear.
1 Like
Thanks! Makes sense. With other customers with many variants we have seen customers do a few different things:
-
Rather than trying to have 1 top level test plan, break it into a few to reduce the number of parameters that change. Then a top level Test Step could accept those inputs and determine which Test Plan to load. @ivan.diep is working on something similar to this.
-
Rather than have each parameter broken out, something more like profile or setup file could be used. Again the 1st Test Step in your plan could load this. Your you could create a custom setting and/or custom panel. Ever Test Step has access to the settings:
@rolf_madsen and others may have some different ideas as well.
Sounds good. We will still look in to the performance side and see if there are ways to improve it. But let us know if you end up trying any of the above options (or something new!). We are happy to help.
Thank you for all the tips, I’ll try and update this post if I found any good practice.
I still think passing parameters is the best way to do if there is no performance issue.
It brings a great benefit of we can see the value directly in the TestPlan, other options seems doesn’t provide such transparency.
At this moment, I’m heavily relies on the parameter from test suite to enable and disable the bottom level steps. Because we have different number of DUTs in each bench.
If I move the control logic of number of DUTs to top level, I feel the TestPlan become harder to maintain with lot of duplicated code, especially in bigger TestPlans.
1 Like
