MongoDB anyone?

@carlos.montes, thanks for suggesting this, and I will try to find the time to keep updating the topic.

I initially thought of posting the ResultListener verbatim, but due to my result storage paradigm, I decided it may be better to begin with a walkthrough of sorts, to get interested readers thinking about the whole result lifecycle.
I don’t want to force my opinions on anyone about how results should be stored, so it may be worth starting from the beginning to create the ResultListener from scratch. We can also discuss the fine things about results because I don’t think there is one size that fits all.

In the following posts, I will share some pretty straightforward code to create an operational MongoDB ResultListener, but before that, there are some things we need to have in place first.

4 Likes

Some of the first great things about MongoDB that grabbed me were:

  1. Community Server
  2. It can be run locally with almost zero configuration.

So, go grab a copy from https://www.mongodb.com/try/download/community.
Currently I think I’m running 4.2.15 and this does everything I need. Don’t bother installing all the extra stuff as you’ll only need the server. Compass is there free backend admin gui, but I never got on with it in the early days, so i never went back. I use the paid Studio 3T for everything but it can all be done on the commandline.

Before we really get into things lets me set some expectations.

  • Installation and configuration is straightforward. The install ask if you want to set it up as a service (recommended).
  • Writing queries would be almost impossible without Studio 3T.
  • It’s not SQL, Databinding to gui controls isn’t like using entity frameworks. Third party developers may hve produced ‘connectors’ by now though.
  • You will have to create your own reporting app to get the data.
3 Likes

The simplest way to demonstrate MongoDB is to create it within the Developer’s Systema and use the Example.sln.

  • Created a new class file in the PluginDevelopment project under the ResultListeners folder.
  • Manage NuGet packages and search and add MongoDB packages as shown below. I may be using older code because it asked me to include a reference to mongocsharpdriver :man_shrugging:.
  • You may also be asked to add a reference to Microsoft.Sharp and System.Core.

image

I’ll post some code later. I’ve made it pretty simple and included everything so it fits in a single .cs file.
I’ll post as a code block but it’s gonna be a few hundred lines.

2 Likes
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;
using MongoDB.Driver.GridFS;

namespace OpenTap.Plugins.PluginDevelopment.ResultListeners
{
    [Display("MongoDB Result Listener", Group: "Database", Description: "Pushes results to a MongoDB instance.")]
    public class MongoDbResultListener : ResultListener
    {
        [Display("Connection String", Group: "Connection", Order: 10.01, Description: "mongodb://127.0.0.1 | mongodb://mongodb0.example.com:27017")]
        public string ConnectionString { get; set; }

        [Display("Test Connection")]
        [Browsable(true)]
        public void TestConnection()
        {
            OnActivity();
            Open_Impl();
            Close();
        }

        private Dictionary<Guid, MongoTestStepRun> _testStepRuns;
        private readonly Dictionary<string, ObjectId> _resultList = new Dictionary<string, ObjectId>();

        private MongoTestPlanRun _testPlanRun;
        private Db _db;

        public MongoDbResultListener()
        {
            Name = "MongoDB";
            ConnectionString = "mongodb://127.0.0.1:27017";
        }


        /// <summary>
        /// Stores the TestPlanRun instance and writes the data to the database.
        /// </summary>
        /// <param name="planRun"></param>
        public override void OnTestPlanRunStart(OpenTap.TestPlanRun planRun)
        {

            _testStepRuns = new Dictionary<Guid, MongoTestStepRun>();

            var pDutId = planRun.Parameters.Where(x => x.MacroName != null && x.MacroName.Equals("DUT ID", StringComparison.CurrentCultureIgnoreCase));
            var dutId = pDutId.Any() && string.IsNullOrWhiteSpace(pDutId.First().Value.ToString()) == false ? pDutId.First().Value.ToString().Trim() : "Default";

            EngineSettings.Current.OperatorName = System.Security.Principal.WindowsIdentity.GetCurrent().Name;
            EngineSettings.Current.StationName = Environment.MachineName;

            _testPlanRun = new MongoTestPlanRun
            {
                _id = ObjectId.GenerateNewId(),
                Guid = planRun.Id,
                TestPlanName = planRun.TestPlanName,
                StartTime = planRun.StartTime,
                StartTimeStamp = planRun.StartTimeStamp,
                TestStartTimezoneOffset = TimeZone.CurrentTimeZone.GetUtcOffset(planRun.StartTime),
                Duration = new TimeSpan(),
                TestPlanXml = planRun.TestPlanXml,
                Verdict = planRun.Verdict.ToString(),
                FailedToStart = planRun.FailedToStart,
                Parameters2 = planRun.Parameters.ToDictionaryOfParameters(),
                Parameters = planRun.Parameters.ToListOfParameters(),
                DutId = dutId,
                User = EngineSettings.Current.OperatorName,
                Station = EngineSettings.Current.StationName
            };

            _db.AddTestPlanRun(_testPlanRun);
            Log.Info($"Test plan '{planRun.TestPlanName}' started");

        }

        /// <summary>
        /// Stores the TestStepRun instance and writes the data to the database.
        /// </summary>
        /// <param name="stepRun"></param>
        public override void OnTestStepRunStart(TestStepRun stepRun)
        {
            base.OnTestStepRunStart(stepRun);

            _resultList.Clear();

            var newTestStepRun = new MongoTestStepRun
            {
                _id = ObjectId.GenerateNewId(),
                TestPlanRunId = _testPlanRun._id,
                TestPlanRunGuid = _testPlanRun.Guid,
                Guid = stepRun.Id,
                Parent = stepRun.Parent,
                SuggestedNextStep = stepRun.SuggestedNextStep,
                SupportsJumpTo = stepRun.SupportsJumpTo,
                TestStepId = stepRun.TestStepId,
                TestStepName = stepRun.TestStepName,
                TestStepTypeName = stepRun.TestStepTypeName,
                Duration = stepRun.Duration,
                Parameters2 = stepRun.Parameters.ToDictionaryOfParameters(),
                Parameters = stepRun.Parameters.ToListOfParameters(),
                StartTime = stepRun.StartTime,
                StartTimeStamp = stepRun.StartTimeStamp,
                StartTimeZoneOffset = TimeZone.CurrentTimeZone.GetUtcOffset(stepRun.StartTime),
                Verdict = stepRun.Verdict.ToString()
            };

            _testStepRuns.Add(stepRun.Id, newTestStepRun);

            _db.AddTestStepRun(newTestStepRun);


        }

        /// <summary>
        /// Writes the TestResult table to the database.
        /// </summary>
        /// <remarks>
        /// Single row tables with multiple columns are scalar and treated as a new published result that
        /// transforms all columns into parameters.
        /// Calling Publish or PublishTable with multiple columns and rows defines the ResultTable structure.
        /// Images, arrays and data tables are stored in GridFS blobs.
        /// Calling PublishTable without first calling Publish will result in an error.
        /// If a column is called PathToFile then the string value of the column is used to
        /// get a file from the local machine to upload to the database.
        /// If the  only column is PathToFile then it is assumed to be an update to an existing result.
        /// </remarks>
        /// <param name="stepRun"></param>
        /// <param name="result"></param>
        public override void OnResultPublished(Guid stepRun, ResultTable result)
        {
            const string fileColumnName = "PathToFile";
            const string fileTitleColumnName = "FileTitle";

            try
            {
                OnActivity();

                if (!_testStepRuns.TryGetValue(stepRun, out var testStepRun))
                {
                    Log.Info($"Error storing test step run for {stepRun}.");
                    return;
                }

                var hasFile = result.Columns.Any(x => x.Name.Equals(fileColumnName));
                var isArray = result.Rows > 1;

                var resId = ObjectId.GenerateNewId();
                if (_resultList.ContainsKey(result.Name))
                    _resultList[result.Name] = resId;
                else
                    _resultList.Add(result.Name, resId);


                if (isArray)
                {
                    // Data is an array.
                    // The data will be stored as a blob in GridFS.
                    var dataTable = result.Columns.ToDictionary(column => column.Name, column => column.Data.Cast<object>().ToList());

                    var tableId = ObjectId.GenerateNewId();
                    _db.SaveDataTableAsync(tableId, resId, dataTable, null);
                }

                // If the results has an image.
                if (hasFile)
                {
                    var pathToFile = result.Columns.FirstOrDefault(x => x.Name.Equals(fileColumnName));
                    var filetTitle = result.Columns.FirstOrDefault(x => x.Name.Equals(fileTitleColumnName));
                    var imageId = ObjectId.GenerateNewId();
                    _db.SaveFileObjectAsync(imageId, resId, pathToFile?.GetValue<string>(0), filetTitle?.GetValue<string>(0));

                }

                // Data is scalar.
                // Each column becomes a parameter and stored with the results.
                // Do not store the path to file.
                var additionalParameters = new ResultParameters();
                foreach (var col in result.Columns)
                {
                    if (!col.Name.Equals(fileColumnName) && !col.Name.Equals(fileTitleColumnName))
                        additionalParameters.Add(new ResultParameter("Result", col.Name, (IConvertible)col.Data.GetValue(0)));
                }

                // This is important as subsequent results published inside the same TestStepRun accumulate parameters.
                var mongoParams = new MongoParameter[testStepRun.Parameters.Count];
                testStepRun.Parameters.CopyTo(mongoParams);

                var resultData = new MongoTestResult
                {
                    _id = resId,
                    TestPlanRunId = testStepRun.TestPlanRunId,
                    TestPlanRunGuid = testStepRun.TestPlanRunGuid,
                    Guid = testStepRun.Guid,
                    Parent = testStepRun.Parent,
                    Name = result.Name,
                    Parameters = mongoParams.ToList(),
                };

                resultData.Parameters.AddRange(additionalParameters.ToListOfParameters());
                resultData.Parameters2 = resultData.Parameters.ToDictionaryOfParameters();

                _db.AddStepResult(resultData);

            }
            catch (Exception ex)
            {
                Log.Error($"Failed to store result for {Name}");
                Log.Error($"{ex.Message}");
            }
        }

        /// <summary>
        /// Updates the TestStepRun after completion.
        /// </summary>
        /// <param name="stepRun"></param>
        public override void OnTestStepRunCompleted(TestStepRun stepRun)
        {
            base.OnTestStepRunCompleted(stepRun);

            if (!_testStepRuns.TryGetValue(stepRun.Id, out var testStepRun))
            {
                Log.Info($"Error updating test step run for {stepRun}.");
                return;
            }

            testStepRun.Duration = stepRun.Duration;
            testStepRun.Verdict = stepRun.Verdict.ToString();

            _db.UpdateTestStepRun(testStepRun);

            _testStepRuns.Remove(stepRun.Id);
        }

        /// <summary>
        /// Updates the TestPlanRun after completion.
        /// </summary>
        /// <param name="planRun"></param>
        /// <param name="logStream"></param>
        public override void OnTestPlanRunCompleted(OpenTap.TestPlanRun planRun, Stream logStream)
        {

            _testPlanRun.Duration = planRun.Duration;
            _testPlanRun.Verdict = planRun.Verdict.ToString();
            _testPlanRun.FailedToStart = planRun.FailedToStart;
            _testPlanRun.LogStreamId = ObjectId.GenerateNewId();

            _db.UpdateTestPlanRun(_testPlanRun);
            _db.SaveStreamAsync(_testPlanRun.LogStreamId, _testPlanRun._id, $"{_testPlanRun.DutId} {_testPlanRun.StartTime.ToString("yyyyMMddTHHmmss")}.txt", logStream);
        }

        public override void Open()
        {
            Open_Impl();
            base.Open();
        }

        public bool Open_Impl()
        {
            string dbName = "Demo1";

            _db = new Db();

            var success = _db.Connect(ConnectionString, dbName);

            if (success)
                Log.Info($"{Name} connection OK.");
            else
                Log.Error($"{Name} connection FAILED.");


            return success;

        }

    }

    public class Db
    {
        private const string CollectionTestPlanRun = "TestPlanRuns";
        private const string CollectionTestResults = "TestResults";
        private const string CollectionTestStepRuns = "TestStepRuns";
        private const string BucketImages = "Images";
        private const string BucketTables = "Tables";
        private const string BucketLogs = "Logs";

        public IMongoClient Client { get; private set; }

        public IMongoDatabase Database { get; private set; }

        public TimeSpan ServerTimeout { get; set; } = new TimeSpan(0, 0, 0, 0, 30000);

        /// <summary>
        /// Connect to the serve.
        /// </summary>
        /// <param name="connectionString"></param>
        /// <param name="databaseName"></param>
        /// <returns></returns>
        public bool Connect(string connectionString, string databaseName)
        {
            Client = new MongoClient(connectionString);

            // All this does is create the database object in the API.
            // It not create a database on the server. MondoDB will only create the database when
            // there is information to write to it.
            Database = Client.GetDatabase(databaseName);

            // Grab a list databases to confirm we have connected.
            var dbs = Client.ListDatabaseNames();

            var dbNames = new List<string>();
            dbs.ForEachAsync(x => dbNames.Add(x));

            return dbNames.Count > 0;

        }

        /// <summary>
        /// Insert TestPlanRun.
        /// </summary>
        /// <param name="testPlanRun"></param>
        public async void AddTestPlanRun(MongoTestPlanRun testPlanRun)
        {
            await InsertOneAsync(Database.GetCollection<MongoTestPlanRun>(CollectionTestPlanRun), testPlanRun).ConfigureAwait(false);
        }

        /// <summary>
        /// Insert TestStepRun
        /// </summary>
        /// <param name="testStepRun"></param>
        public async void AddTestStepRun(MongoTestStepRun testStepRun)
        {
            await InsertOneAsync(Database.GetCollection<MongoTestStepRun>(CollectionTestStepRuns), testStepRun).ConfigureAwait(false);
        }

        /// <summary>
        /// Insert TestResult.
        /// </summary>
        /// <param name="testResult"></param>
        public async void AddStepResult(MongoTestResult testResult)
        {
            await InsertOneAsync(Database.GetCollection<MongoTestResult>(CollectionTestResults), testResult).ConfigureAwait(false);
        }

        /// <summary>
        /// Insert a one document.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="entity"></param>
        /// <param name="data"></param>
        /// <returns></returns>
        private async Task InsertOneAsync<T>(IMongoCollection<T> entity, T data)
        {
            try
            {
                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    await entity.InsertOneAsync(data, null, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }
        }

        public async void UpdateTestStepRun(MongoTestStepRun testStepRun)
        {
            var builder = Builders<MongoTestStepRun>.Filter;
            var filter = builder.Eq(x => x._id, testStepRun._id);

            var update = Builders<MongoTestStepRun>.Update
                .Set(x => x.Parameters2, testStepRun.Parameters2)
                .Set(x => x.Duration, testStepRun.Duration)
                .Set(x => x.Verdict, testStepRun.Verdict);

            try
            {

                var entity = Database.GetCollection<MongoTestStepRun>(CollectionTestStepRuns);
                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    var updateResult = await entity.UpdateOneAsync(filter, update, null, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }
        }

        /// <summary>
        /// Update the Test Plan Run.
        /// </summary>
        /// <param name="testRun"></param>
        public async void UpdateTestPlanRun(MongoTestPlanRun testRun)
        {
            var builder = Builders<MongoTestPlanRun>.Filter;
            var filter = builder.Eq(x => x._id, testRun._id);

            var update = Builders<MongoTestPlanRun>.Update
                .Set(x => x.Duration, testRun.Duration)
                .Set(x => x.Verdict, testRun.Verdict)
                .Set(x => x.FailedToStart, testRun.FailedToStart)
                .Set(x => x.LogStreamId, testRun.LogStreamId);

            try
            {

                var entity = Database.GetCollection<MongoTestPlanRun>(CollectionTestPlanRun);
                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    await entity.UpdateOneAsync(filter, update, null, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }
        }

        /// <summary>
        /// Save an image file to GridFS using the path to the file.
        /// </summary>
        /// <param name="gfsId"></param>
        /// <param name="resultId"></param>
        /// <param name="filePath"></param>
        /// <param name="title"></param>
        /// <param name="type"></param>
        public async void SaveFileObjectAsync(ObjectId gfsId, ObjectId resultId, string filePath, string title, string type = "Image")
        {
            if (string.IsNullOrWhiteSpace(filePath)) return;
            if (!File.Exists(filePath)) return;

            if (string.IsNullOrWhiteSpace(title))
                title = "Not Set";

            var fn = Path.GetFileName(filePath);

            var metaData = new Dictionary<string, object>
            {
                { "ResultId", resultId },
                { "Title", title },
                { "DataType", type }
            };

            var gridFsUploadOptions = new GridFSUploadOptions { Metadata = new BsonDocument(metaData) };

            try
            {
                var bucket = new GridFSBucket(Database, new GridFSBucketOptions
                {
                    BucketName = BucketImages,
                });

                var source = File.ReadAllBytes(filePath);
                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    await bucket.UploadFromBytesAsync(gfsId, fn, source, gridFsUploadOptions, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }

        }

        /// <summary>
        /// Stores array data as a byte in GridFs.
        /// </summary>
        /// <param name="gfsId"></param>
        /// <param name="resultId"></param>
        /// <param name="data"></param>
        /// <param name="title"></param>
        /// <param name="type"></param>
        public async void SaveDataTableAsync(ObjectId gfsId, ObjectId resultId, Dictionary<string, List<object>> data, string title, string type = "DataTable")
        {
            if (data == null) return;
            var bsonByte = data.ToBson();

            if (bsonByte.LongLength == 0) return;

            if (string.IsNullOrWhiteSpace(title))
                title = "Not Set";

            var metaData = new Dictionary<string, object>
            {
                { "ResultId", resultId },
                { "Title", title },
                { "DataType", type }
            };

            var gridFsUploadOptions = new GridFSUploadOptions
            {
                Metadata = new BsonDocument(metaData)
            };

            try
            {
                var bucket = new GridFSBucket(Database, new GridFSBucketOptions
                {
                    BucketName = BucketTables,
                });

                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    await bucket.UploadFromBytesAsync(gfsId, gfsId.ToString(), bsonByte, gridFsUploadOptions, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }
            catch (Exception)
            {
                // ignored
            }
        }

        /// <summary>
        /// Write the log to the database.
        /// </summary>
        /// <param name="gfsId"></param>
        /// <param name="testPlanRunId"></param>
        /// <param name="fn"></param>
        /// <param name="logStream"></param>
        /// <param name="type"></param>
        public async void SaveStreamAsync(ObjectId gfsId, ObjectId testPlanRunId, string fn, Stream logStream, string type = "Stream")
        {
            if (logStream == null) return;

            var metaData = new Dictionary<string, object>
            {
                { "TestPlanRunId", testPlanRunId },
                { "DataType", type }
            };

            var gridFsUploadOptions = new GridFSUploadOptions { Metadata = new BsonDocument(metaData) };

            try
            {
                var bucket = new GridFSBucket(Database, new GridFSBucketOptions
                {
                    BucketName = BucketLogs,
                });

                using (var timeoutCancellationTokenSource = new CancellationTokenSource(ServerTimeout))
                {
                    await bucket.UploadFromStreamAsync(gfsId, fn, logStream, gridFsUploadOptions, timeoutCancellationTokenSource.Token).ConfigureAwait(false);
                }
            }
            catch (OperationCanceledException ex)
            {

            }
            catch (Exception ex)
            {
                // ignored
            }
        }

    }

    public class MongoTestPlanRun
    {
        public ObjectId _id { get; set; }
        public Guid Guid { get; set; }
        public string TestPlanName { get; set; }
        public string TestPlanXml { get; set; }
        public string User { get; set; }
        public string Station { get; set; }
        public DateTime StartTime { get; set; }
        public long StartTimeStamp { get; set; }
        public TimeSpan TestStartTimezoneOffset { get; set; }
        public TimeSpan Duration { get; set; }
        public string Verdict { get; set; }
        public bool FailedToStart { get; set; }
        public Dictionary<string, MongoParameter> Parameters2 { get; set; }
        [BsonIgnore]
        public List<MongoParameter> Parameters { get; set; }
        public string DutId { get; set; }
        public ObjectId LogStreamId { get; set; }

    }

    public class MongoTestStepRun
    {
        public ObjectId _id { get; set; }
        public ObjectId TestPlanRunId { get; set; }
        public Guid TestPlanRunGuid { get; set; }
        public Guid Guid { get; set; }
        public Guid Parent { get; set; }
        public Guid? SuggestedNextStep { get; set; }
        public bool SupportsJumpTo { get; set; }
        public Guid TestStepId { get; set; }
        public string TestStepName { get; set; }
        public string TestStepTypeName { get; set; }
        public DateTime StartTime { get; set; }
        public long StartTimeStamp { get; set; }
        public TimeSpan StartTimeZoneOffset { get; set; }
        public TimeSpan Duration { get; set; }
        public string Verdict { get; set; }
        public Dictionary<string, MongoParameter> Parameters2 { get; set; }
        [BsonIgnore]
        public List<MongoParameter> Parameters { get; set; }
    }

    public class MongoTestResult
    {
        public ObjectId _id { get; set; }
        public ObjectId TestPlanRunId { get; set; }
        public Guid TestPlanRunGuid { get; set; }
        public Guid Guid { get; set; }
        public Guid Parent { get; set; }
        public string Name { get; set; }
        public Dictionary<string, MongoParameter> Parameters2 { get; set; }
        [BsonIgnore]
        public List<MongoParameter> Parameters { get; set; }
    }

    /// <summary>
    /// Represents the OpenTAP Parameter class for MongoDB.
    /// </summary>
    [BsonDiscriminator("Parameter")]
    public class MongoParameter
    {
        public string Name { get; set; }
        public string Group { get; set; }
        public int ParentLevel { get; set; }
        public string MacroName { get; set; }
        public bool IsMetaData { get; set; }
        public dynamic Value { get; set; }
    }

    internal static class TapExtension
    {
        /// <summary>
        /// Transform OpenTAP ResultsParameters in to a dictionary of parameters.
        /// This will create a Parameters BSON document containing named parameter documents.
        /// </summary>
        /// <param name="tapParams"></param>
        /// <returns></returns>
        internal static Dictionary<string, MongoParameter> ToDictionaryOfParameters(this ResultParameters tapParams)
        {
            var mongoParams = new Dictionary<string, MongoParameter>();
            var n = 1;
            foreach (var rp in tapParams.OrderBy(x => x.Name))
            {
                var cleanedKey = $"{rp.Group?.Replace(".", "_")}{rp.Name.Replace(".", "_")}";

                if (tapParams.Count(x => x.Name.Equals(rp.Name)) > 1)
                    cleanedKey += $"_{n++}";
                else
                    n = 1;

                if (mongoParams.ContainsKey(cleanedKey)) continue;

                var newParam = new MongoParameter
                {
                    Name = rp.Name,
                    Group = string.IsNullOrWhiteSpace(rp.Group) ? "System" : rp.Group,
                    Value = rp.Value,
                };

                mongoParams.Add(cleanedKey, newParam);
            }
            return mongoParams;
        }

        /// <summary>
        /// Transform a list of MongoParameters in to a dictionary of parameters.
        /// This will create a Parameters BSON document containing named parameter documents.
        /// </summary>
        /// <param name="tapParams"></param>
        /// <returns></returns>
        internal static Dictionary<string, MongoParameter> ToDictionaryOfParameters(this List<MongoParameter> tapParams)
        {
            var mongoParams = new Dictionary<string, MongoParameter>();
            var n = 1;
            foreach (var rp in tapParams.OrderBy(x => x.Name))
            {
                var cleanedKey = $"{rp.Group?.Replace(".", "_")}{rp.Name.Replace(".", "_")}";

                if (tapParams.Count(x => x.Name.Equals(rp.Name)) > 1)
                    cleanedKey += $"_{n++}";
                else
                    n = 1;

                if (mongoParams.ContainsKey(cleanedKey)) continue;

                var newParam = new MongoParameter
                {
                    Name = rp.Name,
                    Group = string.IsNullOrWhiteSpace(rp.Group) ? "System" : rp.Group,
                    Value = rp.Value,
                };

                mongoParams.Add(cleanedKey, newParam);
            }
            return mongoParams;
        }

        /// <summary>
        /// Transform OpenTAP ResultsParameters in to a list of parameters.
        /// This will create a Parameters BSON document containing a list of parameter documents.
        /// </summary>
        /// <param name="tapParams"></param>
        /// <returns></returns>
        internal static List<MongoParameter> ToListOfParameters(this ResultParameters tapParams)
        {
            return tapParams.OrderBy(x => x.Name)
                .Select(rp => new MongoParameter
                {
                    Name = rp.Name,
                    Group = string.IsNullOrWhiteSpace(rp.Group) ? "System" : rp.Group,
                    Value = rp.Value,
                    //IsMetaData = rp.IsMetaData,
                    //ParentLevel = rp.ParentLevel
                }).ToList();
        }
    }
}

2 Likes

This is really useful @jason.hicks . And you tried to tell me you didn’t have anything you thought you’d be able to contribute for a result listener :wink:

Can you share a little bit about why you chose Mongo DB vs PostgreSQL or other SQL databases?

Hahahhaahaha!

It was just that this is so different from the other implementations that I felt it really wans’nt going to be much use to anyone. It will not work with the Results Viewer and I had to create my own app to use the data.

I had a top-down approach to the whole results lifecycle. It was easy to generate test steps and produce masses of data. It’s easy to export data to Excel for processing of some sort, but the difficulty was creating tabulated data reports and at speed and without too much user input. Once I’d come up with a data model that was flexible enough to accommodate all types of data it was just a question of designing the reporting and query mechanism to access it.

As the designer of the test step, I control how it should be viewed, say a Word document. It is still exportable to Excel when engineers want to look at the data in more detail to reshape and analyse it in some other way.

I have a very lightweight reporting app that uses predefined queries to extract and shape the data into a report. I design the step, the result fields and the query. With four clicks and within a few seconds I can have a few-hundred-page report of tabulated data and images. As an example, a parent step loops over many modes of operation (protocols, BW’s, data rates and channels), a child step may measure the occupied bandwidth fore mode, save the image from the analyser and collect the trace data and settings. All this is stored against an individual result and is retrieved by the reporting app.

I used to love SQL databases, but I was never happy about the data models needed for test result data. It was always as if it was designed so data could be stored super-efficiently and for the purpose of the test system. MongoDB changes all that.
The collections have a schema so we can easily and quickly find stuff and put stuff in. However, as long the BSON document meets this, the rest of it can be anything else. It simplifies my life massively.
In my years of experience, the biggest issue has always been the data and I’ve generally disagreed with everyone about it. Not that I’m right, just that there what I ever seen or used doesn’t seem to allow me to do what I want it to do.

The example ResultListener will allow very large arrays (30K rows x 20 columns +), images (or any file type) and simple scalar results with many fields. It will have some significant differences with how PostgreSQL stores data as I don’t have any visibility of what happening when publishing hooks are invoked. that said, my implementation does what I need it to do.
I tried the example with a couple of the publishing example steps and it seemed to show the right stuff. You can see the name and the result values are there (I checked) but they are hidden deeper in the parameter objects. You can also see them in my reporting app and resulting Excel export.

image

2 Likes

What the test step could post information via the ResultListener how it could be presented?
That way the designer would have the majority control of the whole result lifecycle from cradle to grave.

1 Like

Thanks @jason.hicks this makes a lot of sense. I actually don’t see this not being able to work with the Result Viewer as a negative/limitation. We fully recognize other ways to store and visualize data will be needed. And getting data into MongoDB allows for other tools to interact directly with the DB for visualization. If the Result Viewer was still needed it would always be possible to also turn on the SQLite DB (although at the amount of data you are talking about it would not be that efficient).

I think we could eventually take this and put it on Gitlab. Even if only as an example people could tweak.

@carlos.montes does the above address the questions you had?

1 Like

It should be noted that the raw BSON document Parameters2 fields are the same are the OpenTAP ResultParameter. However, the name is preceeded by either System or Result depending on where it originates. I probably did this to avoid naming conflicts. You can see the Object called ResultVoltage [mv] is really just the field Voltage [mV].

1 Like

One last thing.

The reason arrays, files and logs are stored in GridFS is because of the BSON Document size limitation 16 MB. I decide that there was no time penalty writing and reading to GridFS, and arrays, files and logs would very likely exceed this limit. There is however a storage penalty due to minimum chunck size used in GridFS. But as you will notice, I’m not concerned with storage efficiencies as I store everything for every result. This was intentional to improve the query time. Effectively eliminating SQL subquery and joins, in MongDB speak, allowing for more efficient projections.

An extract of a an occupied bandwidth report taken from the MongoDB datab=base.

1 Like

@jason.hicks This is really helpful and it provides a very good starting point to start making a basic implementation. I copied your code to a project and I glanced through the code and noticed that depending on the column name you define if it is an image, specifically if the column name is “PathToFile”. I also found that if you receive more than 1 row then you treat it as an array. Could you share some of your Publish calls? just want to look at the different cases of Publish.

I will install the nuget packages to give it a try, thanks again for sharing!

2 Likes

@carlos.montes, thanks!

What I’ve posted is obviously missing any error checking and security features, but as you have said, it provides a good starting point and showcases how to use the C# driver.
My own MongoDB ResultListener, which is loosly based on the posted code, uses connection check and pasword security etc. Each test station has a local MongoDB server running as a service. This is an always-on, always-connected results store fallback. There is also an network MongoDB results server. I add two MongoDB listeners to each system, one point locally and the other pointing to the network server.
This is me been over cautious as i have no mechanism yet for redundancy and interuptions.

I think it works correctly for the publishing examples included in Developer’s System, but I post some other examples.

@carlos.montes

  1. PathToFile provides a very simple way to pass info to the publishing engine. @brennen_direnzo, any chance of formalising this and giving me an overloaded call to pass specific types of data.?
  2. Yup, I simply look to see if the data is an array. This is because other fields provided by parent steps should not form part of step publishing the array result. All columns (fields) in the array step form part of the array data-set and parent fields are actually metadata. A simple way to see this is to make a ‘Repeat’ the parent of the ‘Results for XY Array’. You can now see the parent field System-Iteration. The array table is still the same as the one I posted earlier but now I have three of them, one for each iteration. You can see the xl sheet tabs. Each tab is specific to a specific iteration by the _id field.

  1. Because metadata fields cannot be added to the PublishTable without forming part of the array itself I have implemented other controls that allow me to Publish and PublishTable that bind them to the same test result entry. This means I can publish scalar results that have many fields of metadata, but I can also add images and arrays to it. This type of functionality isn’t in the demo code though as it would break all uses of array type data for out-of the box examples. It’s a poor workaround.
1 Like

Replace the Run method in PathAttributeExample and it publishes the file to MongoDB when a file is selected for MyVariousMediaFilesPaths.

        public override void Run()
        {

            var thisStepId = "Image Result";

            var resParams = new ResultParameters
            {
                new ResultParameter(thisStepId, name: "PathToFile", MyVariousMediaFilesPaths),
            };

            Results.Publish(thisStepId, resParams.Select(x => x.Name).ToList(), resParams.Select(x => x.Value).ToArray());
        }

You can see the two files I’ve tried it with in this Studio 3T screenshot. The only thing is that the GridFS bucket is called images, but this should maybe have been called Files.

Studio 3t is actually quite neat because you can access the files directly from within and it will open it in a compatible app.

I ran the XY Array step under a repeat of 3 with 10,000 points with an observable increasem in execution time due to the way the step builds the array.

3 repeat with 10K points.

00:00:00.000000 ; TestPlan       ; Information ; Starting TestPlan 'demo' on 07/16/2021 11:58:59, 2 of 5 TestSteps enabled.
00:00:00.010973 ; TestPlan       ; Debug       ; Saved Test Plan XML [10.6 ms]
00:00:00.022065 ; MongoDB        ; Information ; MongoDB connection OK.
00:00:00.022065 ; MongoDB        ; Information ; Resource "MongoDB" opened. [21.0 ms]
00:00:00.025380 ; MongoDB        ; Information ; Test plan 'demo' started
00:00:00.041339 ; TestPlan       ; Debug       ; PrePlanRun Methods completed [4.70 us]
00:00:00.042336 ; TestPlan       ; Information ; "Repeat" started.
00:00:00.042336 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.053034 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [11.0 ms]
00:00:00.053034 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.054030 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [1.25 ms]
00:00:00.054030 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.056025 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [1.24 ms]
00:00:00.056025 ; TestPlan       ; Information ; "Repeat" completed. [13.8 ms]
00:00:00.056025 ; TestPlan       ; Debug       ; Test step runs finished. [13.9 ms]
00:00:00.056025 ; Summary        ; Information ; ------ Summary of test plan started 07/16/2021 11:58:59 ------
00:00:00.056025 ; Summary        ; Information ;  Repeat                                             13.8 ms         
00:00:00.056025 ; Summary        ; Information ;    Results for XY Array                             11.0 ms         
00:00:00.056025 ; Summary        ; Information ;    Results for XY Array                             1.25 ms         
00:00:00.056025 ; Summary        ; Information ;    Results for XY Array                             1.24 ms         
00:00:00.056025 ; Summary        ; Information ; --------------------------------------------------------------
00:00:00.056025 ; Summary        ; Information ; -------- Test plan completed successfully in 55.6 ms ---------

3 repeat with 1K points

00:00:00.000000 ; TestPlan       ; Information ; Starting TestPlan 'demo' on 07/16/2021 12:58:31, 2 of 5 TestSteps enabled.
00:00:00.010970 ; MongoDB        ; Information ; MongoDB connection OK.
00:00:00.010970 ; MongoDB        ; Information ; Resource "MongoDB" opened. [9.32 ms]
00:00:00.011967 ; TestPlan       ; Debug       ; Saved Test Plan XML [10.6 ms]
00:00:00.017951 ; MongoDB        ; Information ; Test plan 'demo' started
00:00:00.032913 ; TestPlan       ; Debug       ; PrePlanRun Methods completed [4.30 us]
00:00:00.032913 ; TestPlan       ; Information ; "Repeat" started.
00:00:00.032913 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.032913 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [175 us]
00:00:00.032913 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.033909 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [157 us]
00:00:00.033909 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" started.
00:00:00.033909 ; TestPlan       ; Information ; "Repeat \ Results for XY Array" completed. [152 us]
00:00:00.033909 ; TestPlan       ; Information ; "Repeat" completed. [791 us]
00:00:00.033909 ; TestPlan       ; Debug       ; Test step runs finished. [1.13 ms]
00:00:00.041501 ; Summary        ; Information ; ------ Summary of test plan started 07/16/2021 12:58:31 ------
00:00:00.041501 ; Summary        ; Information ;  Repeat                                              791 us         
00:00:00.041501 ; Summary        ; Information ;    Results for XY Array                              175 us         
00:00:00.041501 ; Summary        ; Information ;    Results for XY Array                              157 us         
00:00:00.041501 ; Summary        ; Information ;    Results for XY Array                              152 us         
00:00:00.041501 ; Summary        ; Information ; --------------------------------------------------------------
00:00:00.041501 ; Summary        ; Information ; -------- Test plan completed successfully in 32.9 ms ---------

This is the 1K data extracted with a plot of a noisy sine wave generated using the XY Array step.
Unfortunately, Excel begins to struggle with 10k points in a chart.

1 Like