Test Plan Reference + parametrization: memory / performance issue?

I’m using Test Plan Reference step and plans parametrization a lot, and it seems to me there’s a big memory/performance issue there. I’m not sure whether it’s related to the combination of the two features or just one of them.

Let’s say I have a test plan with 15 exposed parameters. If I do tap run <test plan path> --list-external-parameters -v I can see it takes around 100ms for the parameters to be listed. Now, if create a new test plan, add a Test Plan Reference step that utilizes this plan, and propagate parameters to the newly created plan it already takes around 6s for the parameters to be listed. If the initial test plan also have references to other plans it takes dozens of seconds for the parameters to be listed and just the listing can consume hundreds of megabytes of memory. More complex plans take minutes and gigabytes of memory for the parameters to be listed. These numbers are just to give you an idea.

Listing parameters is not the issue per se, I just find it to be a good indicator of memory/performance issues when parsing/loading test plans which in the end makes it impossible to run them.

Has anyone experienced issues like that?

1 Like

Thanks for this @pythonized it’s probably something either @rolf_madsen or @asger_iversen will have to comment on. What is happening in the background is the test plan is being deserialized into memory, so that may be part of it, but that performance seems poor.

Have you signed the CLA? (CONTRIBUTING.md · master · OpenTAP / OpenTAP · GitLab) If so, you should submit an issue here (https://gitlab.com/OpenTAP/opentap/-/issues/new?issue[milestone_id]=), if not you either can, or I can submit something covering this so that we can track it.

@david-wsd I know you use Test Plan References, do you have similar performance issues?

Is this just a problem with listing external parameters or does it also occur when loading test plans?

@rolf_madsen At the moment it seems to me that the issue is mainly occurs when trying to list parameters. E.g. TUI is not able to show Test Plan Settings even after consuming all the memory it can whereas I can pass a CSV with parameters using -e CLI option and launch the plan.

1 Like

Hi @pythonized, I don’t have any experience running test plans from the command line or TUI, I mainly use the Editor, so my experience is probably not all that relevant.

That said, I don’t currently see any performance issues loading TestPlanReference-heavy plans in the Editor. I just tried loading a test plan with 1000 TestPlanReference steps (with 8 parameters each), it loaded in about 2s, and is reasonably snappy to select steps and see the parameters.

Probably not related, but I did previously experience a performance issue when connecting resources for TestPlanReference steps (see gitlab issue 467), which is now resolved.

2 Likes

@david-wsd Thanks for your input. I think your experience is relevant, because most likely it’s related to OpenTAP itself, not to a specific editor. I’ve just tried Community Edition of the Editor and the behavior is exactly the same.

I do use resources as plan parameters but as I can see from your link it shouldn’t be an issue anymore. Not sure whatI’m doing differently because I cannot even imagine loading 1000 TestPlanReference steps in 2s.

Out of curiosity, are you also using nested TestPlanReference steps? I’m asking because I would love to do that, although for the moment I ensured just a single layer of references to narrow down the issue.

Hi @pythonized, I’m not currently using nested TestPlanReferences. I’ll try that out.

1 Like

@david-wsd I feel really desperate at this point as I cannot achieve any acceptable performance even after removing half of the parameters. A few questions to a fellow TestPlanReference user:

  • Do your parameters have unit attributes?
  • Do you use categories in your parameter names?
  • Do you use multiple references to a same test plan within your high-level test plan?

Hi @pythonized

  • Do your parameters have unit attributes?
    No

  • Do you use categories in your parameter names?
    We usually delete the categories from the parameter name, but sometimes we do use categories.

  • Do you use multiple references to a same test plan within your high-level test plan?
    Yes, we will refer to the same referenced plan hundreds/thousands of times within a single high-level plan.

1 Like

Hi @pythonized, FYI I added a nested TestPlanReference step, so I have a referenced plan within a referenced plan. Loading the 1000-step high-level plan took 10s, so about 5 times longer than without the nested TestPlanReference.

As an experiment, maybe you could try creating a reference plan that has only delay steps. Parameterize a few delays, then make a high-level plan that references it numerous times. Does that also load slowly? If so, that’s an easy test case to share with Keysight.

2 Likes