Is there a way to verify consistency of installed packages and files before running a TestPlan?
In other words, is there a way to know if some of the files in an installation have changed when a TestPlan is run.
A use case of this is ensuring the consistency of files deployed at manufacturing site where we need to ensure none of the files like test limits in a TestPlan are changed intentionally or inadvertently
5 Likes
Hi Nav,
tap package verify can be used to verify the integrity of installed packages.
We used to have an issue on blocking the user from running the test plan in case of any validation errors, maybe you could open an issue on it? I think it is not an uncommon request and it shouldnt be that hard to make with an engine setting.
2 Likes
Hi @rolf_madsen, I read the verify docs, which say it compares the fingerprints of locally installed files to the fingerprints stored in the package. This is a cool feature I hadn’t explored yet. I’m assuming “fingerprint” means hash, correct?
1 Like
Hi @david-wsd, correct, when building the package we calculate a hash for each file so everyone have the chance to verify an installation after it has been deployed.
2 Likes
@rolf_madsen : Since hash calculation already exists, is it possible to integrate that into test plan execution such that when test plan execution is initiated, it can prompt the user and abort the test plan? Or will it require a new feature?
1 Like
If you implement ITestPlanRunMonitor to get an event when the test plan is executed. I think it should be possible to implement this feature there. It could be a bit complicated to figure out which plugins are actually used when the test plan starts, but assuming all of them, it should be easy enough.
Sounds good. Thanks for the information
For me EnterTestPlanRun(TestPlanRun plan) never seems to be called. I would also like to implement a verification of the TestPlan on the begin of the run…
It seems the ITestPlanRunMonitor has to be implemented on a ComponentSettings class. If you need to go this route, probably add Browsable(false) to that class.
Ok, this does work very nicely and is exactly what I was looking for. I don’t know what changed, but indeed Browsable(false) is not required, it works perfectly without it. Thanks!
1 Like
Sure, it should still work. You will get an additional settings tab for that object without Browsable(false).
1 Like
Seeing that I’m not the only one with this need, I post here my code for implementing a TestPlan / Settings verification option. It might not do completely what you want, and it is not 100% checked and fool proof, but it might be valuable for the others here. It adds a Settings tab where you can enable/disable the verification tests. If one of the verifications fails, it aborts the test plan.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.IO;
using System.Text.RegularExpressions;
using OpenTap;
namespace IntegrityVerification
{
[Display("Integrity Verification", Description: "Settings for the integrity verification")]
[Browsable(true)]
public class IntegrityVerification : ComponentSettings<IntegrityVerification>, ITestPlanRunMonitor
{
[Display("Test plan in memory", Description: "Verifies the XML of the test plan in memory with the XML found in the " +
"corresponding test plan on disk.\nThis prevents the test plan to run if there is any discrepancy or if the XML can not be loaded.",
Group: "Integrity Verification", Order: 1.1)]
public bool VerifyTestPlanXml { get; set; }
[Display("Test plans", Description: "Verifies the hash of the all installed .TapPlan's against the hash in the package.xml.\n" +
"If this is enabled and changes to the test plans are detected, the test plan won't run.", Group: "Integrity Verification", Order: 1.2)]
public bool VerifyInstalledTestplanHashes { get; set; }
[Display("Settings", Description: "Verifies the hash of the all installed .xml's against the hash in the package.xml.\n" +
"If this is enabled and changes to the bench settings are detected, the test plan wont run.", Group: "Integrity Verification", Order: 1.3)]
public bool VerifyInstalledSettings { get; set; }
private OpenTap.TraceSource MyLog = OpenTap.Log.CreateSource("IntegrityVerification");
public IntegrityVerification()
{
VerifyTestPlanXml = false;
VerifyInstalledTestplanHashes = false;
VerifyInstalledSettings = false;
}
private bool TestplanXmlCorrespondsToXmlFile(TestPlanRun plan, out string reason)
{
if (plan.Parameters["TestPlanPath"] != null)
{
string path = plan.Parameters["TestPlanPath"] as string;
MyLog.Debug($"Loading {path}");
if (File.Exists(path))
{
string xmlFromFile = File.ReadAllText(path);
string xmlFromMemory = plan.TestPlanXml;
if (xmlFromMemory.Equals(xmlFromFile))
{
reason = "XML in memory is equal to XML found on disk";
return true;
}
else
{
reason = $"XML in memory is NOT equal to XML found on disk ('{path}')";
return false;
}
}
else
{
reason = $"Couldn't load XML file from '{path}'";
return false;
}
}
else
{
reason = "No file associated with test plan. Cannot perform integrity check.";
return false;
}
}
private string GetTapVerifyPackageOutput()
{
System.Diagnostics.Process process = new System.Diagnostics.Process();
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
startInfo.CreateNoWindow = true;
startInfo.FileName = "cmd.exe";
startInfo.Arguments = "/C tap package verify";
startInfo.RedirectStandardOutput = true;
startInfo.UseShellExecute = false;
process.StartInfo = startInfo;
process.Start();
string m = "";
while (!process.HasExited)
{
m += process.StandardOutput.ReadLine() + '\n';
}
return m;
}
private List<string> GetMatches(string input, string reg)
{
var matches = Regex.Matches(input, reg);
List<string> listOfMatches = new List<string>();
foreach (Match p in matches)
{
listOfMatches.Add(p.ToString());
}
return listOfMatches;
}
private List<string> GetAlteredTestPlans(string verificationOutput)
{
string regex = @"(File '.*(?:\.TapPlan)' has non-matching checksum.)";
return GetMatches(verificationOutput, regex);
}
private List<string> GetAlteredBenchSettings(string verificationOutput)
{
string regex = @"(File '.*(?:\.xml)' has non-matching checksum.)";
return GetMatches(verificationOutput, regex);
}
private void AbortIfAnyAltered(TestPlanRun plan, List<string> alteredItems)
{
if (alteredItems.Count > 0)
{
foreach (var u in alteredItems)
{
MyLog.Error(u);
}
MyLog.Error("Test plan execution aborted: one or more checksums did not match. Please reinstall any altered packages to reset the parameters to the initial values.");
plan.MainThread.Abort();
}
}
public void EnterTestPlanRun(TestPlanRun plan)
{
if (VerifyTestPlanXml)
{
string reason;
if (!TestplanXmlCorrespondsToXmlFile(plan, out reason))
{
MyLog.Error($"Test plan execution aborted: {reason}.");
plan.MainThread.Abort();
}
else
{
MyLog.Debug($"XML check passed: {reason}");
}
}
if (VerifyInstalledTestplanHashes | VerifyInstalledSettings)
{
string tapVerificationOutput = GetTapVerifyPackageOutput();
if (VerifyInstalledTestplanHashes)
{
MyLog.Debug("Start verifying installed test plan packages.");
AbortIfAnyAltered(plan, GetAlteredTestPlans(tapVerificationOutput));
}
if (VerifyInstalledSettings)
{
MyLog.Debug("Start verifying installed settings packages.");
AbortIfAnyAltered(plan, GetAlteredBenchSettings(tapVerificationOutput));
}
MyLog.Debug("All plugins verified successfully.");
}
}
public void ExitTestPlanRun(TestPlanRun plan)
{
;
}
}
}
4 Likes