I’m using Test Plan Reference step and plans parametrization a lot, and it seems to me there’s a big memory/performance issue there. I’m not sure whether it’s related to the combination of the two features or just one of them.
Let’s say I have a test plan with 15 exposed parameters. If I do tap run <test plan path> --list-external-parameters -v I can see it takes around 100ms for the parameters to be listed. Now, if create a new test plan, add a Test Plan Reference step that utilizes this plan, and propagate parameters to the newly created plan it already takes around 6s for the parameters to be listed. If the initial test plan also have references to other plans it takes dozens of seconds for the parameters to be listed and just the listing can consume hundreds of megabytes of memory. More complex plans take minutes and gigabytes of memory for the parameters to be listed. These numbers are just to give you an idea.
Listing parameters is not the issue per se, I just find it to be a good indicator of memory/performance issues when parsing/loading test plans which in the end makes it impossible to run them.
Has anyone experienced issues like that?
1 Like
Thanks for this @pythonized it’s probably something either @rolf_madsen or @asger_iversen will have to comment on. What is happening in the background is the test plan is being deserialized into memory, so that may be part of it, but that performance seems poor.
Have you signed the CLA? (CONTRIBUTING.md · master · OpenTAP / OpenTAP · GitLab) If so, you should submit an issue here (https://gitlab.com/OpenTAP/opentap/-/issues/new?issue[milestone_id]=), if not you either can, or I can submit something covering this so that we can track it.
@david-wsd I know you use Test Plan References, do you have similar performance issues?
Is this just a problem with listing external parameters or does it also occur when loading test plans?
@rolf_madsen At the moment it seems to me that the issue is mainly occurs when trying to list parameters. E.g. TUI is not able to show Test Plan Settings even after consuming all the memory it can whereas I can pass a CSV with parameters using -e CLI option and launch the plan.
1 Like
Hi @pythonized, I don’t have any experience running test plans from the command line or TUI, I mainly use the Editor, so my experience is probably not all that relevant.
That said, I don’t currently see any performance issues loading TestPlanReference-heavy plans in the Editor. I just tried loading a test plan with 1000 TestPlanReference steps (with 8 parameters each), it loaded in about 2s, and is reasonably snappy to select steps and see the parameters.
Probably not related, but I did previously experience a performance issue when connecting resources for TestPlanReference steps (see gitlab issue 467), which is now resolved.
2 Likes
@david-wsd Thanks for your input. I think your experience is relevant, because most likely it’s related to OpenTAP itself, not to a specific editor. I’ve just tried Community Edition of the Editor and the behavior is exactly the same.
I do use resources as plan parameters but as I can see from your link it shouldn’t be an issue anymore. Not sure whatI’m doing differently because I cannot even imagine loading 1000 TestPlanReference steps in 2s.
Out of curiosity, are you also using nested TestPlanReference steps? I’m asking because I would love to do that, although for the moment I ensured just a single layer of references to narrow down the issue.
Hi @pythonized, I’m not currently using nested TestPlanReferences. I’ll try that out.
1 Like
@david-wsd I feel really desperate at this point as I cannot achieve any acceptable performance even after removing half of the parameters. A few questions to a fellow TestPlanReference user:
- Do your parameters have unit attributes?
- Do you use categories in your parameter names?
- Do you use multiple references to a same test plan within your high-level test plan?
Hi @pythonized, FYI I added a nested TestPlanReference step, so I have a referenced plan within a referenced plan. Loading the 1000-step high-level plan took 10s, so about 5 times longer than without the nested TestPlanReference.
As an experiment, maybe you could try creating a reference plan that has only delay steps. Parameterize a few delays, then make a high-level plan that references it numerous times. Does that also load slowly? If so, that’s an easy test case to share with Keysight.
2 Likes
Resurrecting the thread as with the release of OpenTAP 9.16 it seems the situation has worsened.
I’m attaching a simple plan that reproduces the issue:
-
tap run HighLevelTestPlan.TapPlan --list-external-parameters -v – takes around 1.5 minutes to list the only parameter of the test plan, with very high memory/CPU usage
- Editor usually crashes if I try to open
HighLevelTestPlan.TapPlan
Could someone from Keysight team take a look?
HighLevelTestPlan.TapPlan (112.5 KB)
TestPlan.TapPlan (144.5 KB)
subTestPlan.TapPlan (15.3 KB)
@kyler.lee or @rolf_madsen can one of you look at this?
@brennen_direnzo, I have been looking into some improvements in this area. Seems there are issues if you have enough parameters in the test plan.
@pythonized, you can try out this build and see if it helps. http://invalid_link___.com
In any case I’d suggest creating an issue on it.
1 Like
@rolf_madsen I don’t seem to be able to use the provided link nor find MR 4942 in Merge requests · OpenTAP / OpenTAP · GitLab
@rolf_madsen Looks like it improves things for my example plan but doesn’t resolve issues with my real plans that are more complex. Should I try to make it more visible with the example plan or the root cause/direction is already clear?
@pythonized, unfortunately I have no idea what then causes the issue in your plans. I can look into it if you can make a test plan(s) that has the problem.
1 Like
@rolf_madsen I’ve adjusted the example that seems to reproduce the issue with the latest version. High-level test plan essentially does the same thing, I just added another layer of subplans and some parametrization. I checked performance by using tap run HighLevelTestPlan.TapPlan --list-external-parameters -v as always. It consumes enormous amount of memory and time, if it even succeeds.
Just as a remark, I see some mapping issues but from my experience fixing them won’t affect the behavior.
Your help will be very much appreciated as it’s kind of a major/blocking issue. Thank you!
HighLevelTestPlan.TapPlan (169.9 KB)
TestPlan.TapPlan (213.7 KB)
subTestPlan.TapPlan (16.4 KB)
subSubTestPlan1.TapPlan (2.6 KB)