Parallel dynamic testplans

First-time OpenTAP user, here! I’m not entirely sure how to go about this complex flow control scenario. Any pointers would be greatly appreciated.

I have a long-running environmental test. To maximise throughput, I’ll be testing up to 20 DUTs in parallel. All DUTs will undergo an initial stage of testing before commencing the long-running environmental stage. This first stage will differ slightly depending on the DUT part number. I’m pretty confident I’ll be able to build these test plans dynamically and load them with the TestPlan Reference Step.

Here’s where it gets complicated. In the event of a DUT failing on the first stage, I’d like to allow the operator to remove the failed DUT and replace it with a new one, which may be of a different type (different test plan) and perform the first stage on that before starting the next stage. The idea being to only start the long-running part when all DUTs have passed the first stage.

This sounds like I’ll be trying to modify the top-level test plan while it’s running, which I know I can’t do.

All DUTs and instruments are connected via a matrix, so I’m assuming that I will indeed need a top-level plan to ensure the instruments can be shared amongst all the DUTs correctly.

Is what I’m trying to do possible/practical in OpenTAP, or am I better off just writing something custom and saving my first outing for something a little simpler?

Hi @benstreek and welcome to the community!

I am sure that what you want to do can be achieved in OpenTAP, but it might require a bit of customization depending on how exactly you want it to work.

First I think you are best off starting with focusing on testing just one DUT. That can be encapsulated as a unit in the form of a test plan file. I think we both agree that this is straight forward.

When you want to scale it up to testing 20 DUTs in parallel, I’d suggest you do a test plan run for each DUT. This way you will get a set of results for each DUT and that makes it a bit easier to review the results. You could also use a TestPlanReference step, but then every DUT would be a part of the same test plan run and I guess that’d be a bit confusing with regards to verdicts.

To encapsulate a run of a test plan you can do something like this:

    public class SubPlanStep : TestStep
    {
        [FilePath]
        public string TestPlanFile { get; set; }
        public override void Run()
        {
            // OverlayComponentSettings causes new instances of all DUTs and instruments to be created.
            // this creates a layer of isolation between the resources used in the test plan and in the 
            // sub plan.
            using (Session.Create(SessionOptions.RedirectLogging | SessionOptions.OverlayComponentSettings))
            {
                // add a new trace listener to get the redirected log.
                var logListener = new TraceListener();
                OpenTap.Log.AddListener(logListener);
                
                var plan = TestPlan.Load(TestPlanFile);
                
                // Now configure DUTS/instruments. These changes goes into the session overlay and does not affect
                // resources outside the session.
                //InstrumentSettings.Current[0].VisaAddress = xyz
                //DutSettings.Current[0].IP = 123.123.123.123
               
                
                var run = plan.Execute();
               // UpgradeVerdict(run.Verdict); // run.Verdict contains the verdict for that DUT.
            }
        }
    }

Obviously you’d end up needing something a bit different, but maybe it gives you a sense of some of the possibilities you have?

Hi, @rolf_madsen . Thanks for your response, really appreciate it.

I take your point about focussing on one DUT at a time, however, I would like an understanding of how the mechanics of this will work before I spend too much time putting tests together.

I’ve had a look at your suggestion and I don’t think it’s quite going to meet my needs but I’m sure that’s just due to my explanation not being terribly clear!

I believe these sub-plans will only be run once per main-plan run. This means when a DUT fails, I’ll need to restart the main-plan (and therefore all sub-plans) in order to restart just that one sub-plan that failed. Please correct me if I’m wrong on that.

My requirement is to have DUTs pause before the long-running stage to allow all DUTs to pass the first stage, even if that means swapping out failed DUTs, before continuing.

My initial thinking was to write a basic test run manager which would make it fairly simple to create TestRuns on the fly using the API. I can keep the logging separate by simply passing different listener instances to the Execute() method. The only problem I’m having is that the TestRuns will just shutdown resources when they’re finished even if they’re in use by other runs. I was hoping your Session.Create() example would help with this but doesn’t seem to. My other thought for this was writing some sort of ConcurrentResourceManager, but I’m not sure if that will result in unexpected side effects.

How many sub-plans you run depends on how your main plan is constructed. It can be any number of runs.

Session.Create will create a new session. When ComponentSettingsOverlay is used, it will load new instances of every resource. So if the DUT class is properly encapsulated this should give you no problems - if they are not , you might have to run the plan in a different process.

How are you connecting to your DUTs?

1 Like

I can create as many instances of a sub-plan as I like, yes, but I have no flexibility as to when/what sub-plans are run. The sub-plans will have to be selected in advance, all DUTs loaded onto the system and then all plans executd at once by running the main plan.

If each DUT take 30 seconds for the operator to load onto the test system and they’re testing 20, if I can start running the plan on DUT1 as soon as it’s loaded, I can have the majority of the test performed on it by the time the operator’s finished loading the other 19 DUTs. If the unit failed, they can swap with another and start the test again. Once all DUTs have successfully passed the first stage, I can begin the long-running test.

I can write something that will manage the individual sub-plans without making them part of a main-plan at all and see what happens. If I run into problems, I’ll let you know.

1 Like