TFS 2015 Build system - Automated tests

In my previous article, I described the Build Agents and today I will describe how to use them to run the NAV tests for you.

But first - why you need to run the tests at all?

Many years ago, when NAV developer wrote some functionality, he first tested it, before he passed it to someone else (probably customer or some consultant). Someone else did another testing, at the end Customer tested the change in his live DB. But everybody was testing just the new functionality. What was the result? Something else, not connected to the change at first glance, was broken. If you were lucky, it ended with some error message. In worst case, you ended with one year of scrambled ledger entries or something similar, because  somebody noticed the problem after long time. Correct this mess and repair lost trust of the customer cost you much. And automated tests are part of the way to not get into these troubles. In Cloud era you need to multiple this mess e.g. 1000 times. And this is really big number.

It means, going to the cloud without automated tests is like jumping from flying airplane without parachute. Some time it is ok, you enjoy the freefall, but then...

TFS and automated tests

Of course, TFS itself support automated testing. Developers using tools like Visual Studio to develop the app have plenty tools and ways how to write the tests. These tests then are processed during the build process and the result of the tests is taken and uploaded as a result of the build. To take the result and upload it as a product of the test is done in build step named "Publish Test Results":

This steps support different test result formats like "VSTests", "JUnit", "NUnit", "XUnit". It means, if you produce file in correct format, it will take it and process it. But how to produce such a file from NAV tests? As each time, easy answer is "PowerShell". Just run the tests, take the result, process them, generate file in correct format and you are done.

How to run NAV Tests on TFS

Ok, first we need to run the tests. Again, using the powershell it is not so hard. You need to run the Desktop client in console mode (what? Console mode? Yes, to not have all the thing around, just running tests...) and run the test runner codeunit, e.g. like this:

        & $RoleTailoredClientExePath -consolemode -showNavigationPage:0 -settings:"$ConfigFile" "dynamicsnav://$NAVServerName/$NAVServerInstance/$CompanyName/RunCodeunit?Codeunit=$CodeunitId"

This could be just part of your PowerShell script:

After all is finished (could be around 5 hours, depends on hardware, number of tests etc.), you have NAV DB with the test results. Do not forget, that we have run the build agent in console, not as a service, to be able to display the dialog boxes etc.

How to transform results from NAV into Test results

Now, we need to read the data from SQL (or if you prefer webservices, you can use them, e.g. read them through OData or SOAP, it is on you...) and create the correct file. I have decided to create VSTest result file, because I have found some examples and there is XSD installed with the visual studio for the format, thus I can validate the format. Because it is not possible to describe the format here, just look into the Save-NAVTestResultTrx function in Test-NAVDatabase.ps1 script. I was trying to put the available info from NAV into the result as posible. Test names are generated as <CodeunitID>:<TestName>.

During the build process you need to place the file somewhere on the disc and set the Publish Test Result step to take it. How it is named doesn't matter, the step just must find it.

Build results with tests

When all succeeded, you will see the test results on the build information page (screenshot is taken from the latest RC version of TFS "15", which has many nice features in the test area):



You can see there how many tests failed (together with increment/decrement in comparison with previous build), how long it takes to generate them etc.

From there, you can just go to details, where you can list the tests, look at the details, create bug work items for them, set some attributes to categorize the errors etc.:


Now, when you have the tests, you can just go through them and solve the "problem". You can fix the test, or you can fix your code, or may be you just fix some data. Depends on where the source is.

All the data together with data about which change is part of which build gives you really quick pointer to "what goes wrong".

Tips for extending the process

Of course, there are many ways how to make the process much better. E.g. you do not need to run all tests each time, but only the tests covering modified objects. You can do that through using the Test Coverage Map. It is on you, all is in your scripts and C/AL code. What is good for your business, you shall do.


Hope that this serial of articles helped you to prepare your own environment to do what you need, and may be see you on some sessions about TFS on NAVTechDays 2016 in Antwerpen!


I wish you happy testing and many green builds!