First-time OpenTAP user, here! I’m not entirely sure how to go about this complex flow control scenario. Any pointers would be greatly appreciated.
I have a long-running environmental test. To maximise throughput, I’ll be testing up to 20 DUTs in parallel. All DUTs will undergo an initial stage of testing before commencing the long-running environmental stage. This first stage will differ slightly depending on the DUT part number. I’m pretty confident I’ll be able to build these test plans dynamically and load them with the TestPlan Reference Step.
Here’s where it gets complicated. In the event of a DUT failing on the first stage, I’d like to allow the operator to remove the failed DUT and replace it with a new one, which may be of a different type (different test plan) and perform the first stage on that before starting the next stage. The idea being to only start the long-running part when all DUTs have passed the first stage.
This sounds like I’ll be trying to modify the top-level test plan while it’s running, which I know I can’t do.
All DUTs and instruments are connected via a matrix, so I’m assuming that I will indeed need a top-level plan to ensure the instruments can be shared amongst all the DUTs correctly.
Is what I’m trying to do possible/practical in OpenTAP, or am I better off just writing something custom and saving my first outing for something a little simpler?
Hi @benstreek and welcome to the community!
I am sure that what you want to do can be achieved in OpenTAP, but it might require a bit of customization depending on how exactly you want it to work.
First I think you are best off starting with focusing on testing just one DUT. That can be encapsulated as a unit in the form of a test plan file. I think we both agree that this is straight forward.
When you want to scale it up to testing 20 DUTs in parallel, I’d suggest you do a test plan run for each DUT. This way you will get a set of results for each DUT and that makes it a bit easier to review the results. You could also use a TestPlanReference step, but then every DUT would be a part of the same test plan run and I guess that’d be a bit confusing with regards to verdicts.
To encapsulate a run of a test plan you can do something like this:
public class SubPlanStep : TestStep
{
[FilePath]
public string TestPlanFile { get; set; }
public override void Run()
{
// OverlayComponentSettings causes new instances of all DUTs and instruments to be created.
// this creates a layer of isolation between the resources used in the test plan and in the
// sub plan.
using (Session.Create(SessionOptions.RedirectLogging | SessionOptions.OverlayComponentSettings))
{
// add a new trace listener to get the redirected log.
var logListener = new TraceListener();
OpenTap.Log.AddListener(logListener);
var plan = TestPlan.Load(TestPlanFile);
// Now configure DUTS/instruments. These changes goes into the session overlay and does not affect
// resources outside the session.
//InstrumentSettings.Current[0].VisaAddress = xyz
//DutSettings.Current[0].IP = 123.123.123.123
var run = plan.Execute();
// UpgradeVerdict(run.Verdict); // run.Verdict contains the verdict for that DUT.
}
}
}
Obviously you’d end up needing something a bit different, but maybe it gives you a sense of some of the possibilities you have?
Hi, @rolf_madsen . Thanks for your response, really appreciate it.
I take your point about focussing on one DUT at a time, however, I would like an understanding of how the mechanics of this will work before I spend too much time putting tests together.
I’ve had a look at your suggestion and I don’t think it’s quite going to meet my needs but I’m sure that’s just due to my explanation not being terribly clear!
I believe these sub-plans will only be run once per main-plan run. This means when a DUT fails, I’ll need to restart the main-plan (and therefore all sub-plans) in order to restart just that one sub-plan that failed. Please correct me if I’m wrong on that.
My requirement is to have DUTs pause before the long-running stage to allow all DUTs to pass the first stage, even if that means swapping out failed DUTs, before continuing.
My initial thinking was to write a basic test run manager which would make it fairly simple to create TestRuns on the fly using the API. I can keep the logging separate by simply passing different listener instances to the Execute() method. The only problem I’m having is that the TestRuns will just shutdown resources when they’re finished even if they’re in use by other runs. I was hoping your Session.Create() example would help with this but doesn’t seem to. My other thought for this was writing some sort of ConcurrentResourceManager, but I’m not sure if that will result in unexpected side effects.
How many sub-plans you run depends on how your main plan is constructed. It can be any number of runs.
Session.Create will create a new session. When ComponentSettingsOverlay is used, it will load new instances of every resource. So if the DUT class is properly encapsulated this should give you no problems - if they are not , you might have to run the plan in a different process.
How are you connecting to your DUTs?
1 Like
I can create as many instances of a sub-plan as I like, yes, but I have no flexibility as to when/what sub-plans are run. The sub-plans will have to be selected in advance, all DUTs loaded onto the system and then all plans executd at once by running the main plan.
If each DUT take 30 seconds for the operator to load onto the test system and they’re testing 20, if I can start running the plan on DUT1 as soon as it’s loaded, I can have the majority of the test performed on it by the time the operator’s finished loading the other 19 DUTs. If the unit failed, they can swap with another and start the test again. Once all DUTs have successfully passed the first stage, I can begin the long-running test.
I can write something that will manage the individual sub-plans without making them part of a main-plan at all and see what happens. If I run into problems, I’ll let you know.
1 Like
I think I’ve got the behaviour I need now. Using the ComponentSettingsOverlay together with the LazyResourceManager allows me to execute two independent TestPlanRuns simultaneously, with each able to access the same instruments.
My custom instruments and test steps will need some further work to make them behave sensibly in Editor, but I’ll leave that for another post in the future.
Thanks for the help, @rolf_madsen
Spoke too soon, unfortunately. After further testing I’m occasionally ending up with testplanA closing the resource before testplanB is done with it.
I’m working round it by keeping track of Open/Close calls within the instrument but it doesn’t really feel like the best way of doing it.
Unfortunately you cannot have multiple test plans running concurrently in the same session and yes, the primary reason for this is resource management.
I am wondering if you did exactly what I showed in the code sample or if you are doing some deriviative of that?
When TestPlan.Load is called inside the new session, new resource instances will be assigned to the test step, while if you load it outside the session you will get the resources from the old session. I am wondering if this is contributing to your issues.
I did try your code sample, which seemed to work in combination with the LazyReasourceManager. As I say, though, timing issues were causing this to fail occasionally, which would only increase as I add more DUTs and TestSteps.
The specific problem is that my instrument resources need to be shared by all 20 DUTs, and at some point, one of the runs will shut down the underlying connections to the instruments.
I suspect that something could be done with a custom resource manager, but I’m not really familiar with the framework yet, so I don’t fancy tackling that right now. The workaround of keeping a count of open connections inside the instrument seems to be doing the job for now, though it does require not using the ComponentSettingsOverlay. Obviously, having the behaviour only work with a specific configuration is far from ideal. Still, it’ll get me going and hopefully, I can come up with something better in the future.
Ah OK, are you sure that you need to use the same instrument instance? Cant you just open 20 connections to the same instrument using 20 different instances?
Well, obviously now that I’ve tried that again, it works…
This points to your earlier question about using your code sample exactly - thought I did, turns out I did not. I’m not loading the plan from a file at the moment but creating it programmatically, so I took that line out, missing the fact that I still need to instantiate the testplan inside the new session. 
1 Like
Yes, it is not super straight forward, but there are actually quite many possibilities with the sessions. You can also make SessionLocal<T> variables, which have unique values per session - Maybe this could be useful to you.
Anyway, maybe the ‘Dynamic Test Plan’ should be a test step bundled with OpenTAP. It could also be possible to add some ‘placeholder’ child test steps, so you can see the execution status in the ‘subtestplan’. Is this something you are interested in, or do you want to keep the sub-plan hidden / isolated?
1 Like
Session-based variables could well be useful, thanks for the heads-up.
For my current use case, I don’t need the test plan to be visible but it would definitely aid future development of tests for this platform if it could all be put together in Editor and saved out to a file.