All around NAV dev and test
Let's confess before I start: getting standard tests to run against your code is by no means rocket science. However, I dare say many peers out there haven't even touched the Test Toolkit, seemingly looking at it as an unbearable threshold. Too much work. Of no relevance to their code. Unknown = unloved (as my mother used to say).
Clearly the majority of the audience I have been speaking to on the Test Automation Suite in the last year and a half, never laid hands on it. I hope I did trigger some of them though, to get started. And if they, and you, still have this cold water fear to get started … this post will show you there's no reason not to. While showing how easy it is, I will also touch on some backgrounds; of the Test Toolkit and test automation in general.
Now what do we need?
Needless to mention that all resources should be based on the same version. In our case we are still running on NAV 2016: code on RTM (build 42815), executables on CU11 (build 46773). And BTW: no direct need for any fancy hardware, virtualized or not. My first steps were done on my laptop!
O yes, this CRONUS database. Why are we using a CRONUS database? Probably like you, many participants in my sessions wondered. Why not run it on our own database, containing our data: setup, master data, ledger entries, …?
One of the principles in testing is to have a common baseline when (re)executing tests. To give the best guarantee that a test is reproducible. A in, B out. Every time again. No data created in between two runs of the same test (set) that results in a different outcome. Well, that's not a ground vast argument for using CRONUS, right?
However, another principle is that each test (set) creates its own data needed during execution. And ideally the full Monty, meaning from scratch. Every time again, making a test (set) independent of any data already residing in the system … ideally.
As with many things in live it's not always as ideal as we would like, or as we say. This also applies to the Test Automation Suite.
Yes, the majority of the tests create their own data on the fly, like customers, sales documents, ledger entries, etc. But some also rely on the data readily available in CRONUS, like Sales & Receivables Setup or VAT Posting Setup, data - configuration data - that is used in most relevant tests as-is. I guess this also relates to the fact that the Test Automation Suite is mainly executing integration tests. Thus needing to use an complex set of data. To avoid to have to recreate each part of this set with each execution of a test (set), I guess, MS has chosen to make use of some data readily available in CRONUS and making the test execution faster.
So on one hand, having test create their own data, it is of no importance in what database we execute them. On the other, as described, there is some dependency on data in CRONUS and so we are using CRONUS for our tests.
Enough about this baseline, for now …
Or at least the test tool should be executing the first test codeunit, 134000 - ERM Apply Sales/Receivables. And of course continue with all the others. In our case this very first run took approx. 1 hour. Which is quite fast as I had ran the standard Test Suite already on standard CRONUS taking approx. 2,5 hours. That it only took an hour was directly due to the fact that only a minority, 23%, of the tests passed and therefor the execution of all tests was much shorter.
But more later, on the results and how we went about getting the failures succeed.
That's it for now and next time don't tell me you still haven't tried to "Run Standard Tests against Your Code".
With our upgrade to NAV 2016 from NAV 2009 R2 many new things came available to us at The Learning Network, formerly known as Van Dijk Educatie. Technical improvements like the performance of the service tier, which was one of the major reasons for wanting to leave behind NAV 2009. But of course, next to a vast number of functional features, also the availability of PowerShell cmdlets, upgrade codeunits and test automation. Those who have been following me one way or another know that the latter subject has my special attention: testing and test automation.
In today's post I would like to make a start in sharing our approach and findings in how we started to use the Test Toolkit as being released by MS on each NAV product DVD since NAV 2016. For that I have setup a kind of framework below that allows me refer back to some of its parts when elaborating on it more in posts to come.
So here we go, fasten your seatbelts and stay tuned. And … do not be afraid to ask.
Our primary goal of this exercise was to setup a test collateral, based on the standard MS application tests, to be used as a regression test suite, by
The basic plan to achieve this was to:
With any endeavor there are always a number of assumptions. Or should I say loads?
Well, basically we had these two, being that all MS tests …
This is the world we live in at The Learning Network:
To continued ...
Looking at this beautiful Spanish scenery, just before a nice early-morning dive into the swimming pool, a perfect moment to reply to your follow-up post on automated testing, Jan.
Thanks for taking the time to answer my questions, Luc! I’ve definitely learned a thing or two, and I’m glad that we apparently agree about most of this stuff. Just a few last responses below…
Good to hear, and thanx too as it it allowed me to look more conscious at this topic.
About #1. “what to test” Like I said, my first few automated tests were unit tests testing the results of validations. That seemed like a good idea at the time, because it allowed the tests to a. be very limited in scope, and b. be very isolated from each other. Would you consider making separate test functions for each (relevant, i.e., sufficiently complex) field validation, like you probably would for each (relevant) function, or is that the wrong scale as far as you are concerned?
Like I said, my first few automated tests were unit tests testing the results of validations. That seemed like a good idea at the time, because it allowed the tests to a. be very limited in scope, and b. be very isolated from each other.
Would you consider making separate test functions for each (relevant, i.e., sufficiently complex) field validation, like you probably would for each (relevant) function, or is that the wrong scale as far as you are concerned?
My first thought is, that at this very moment it is too far off what we (still) need to achieve in our organisation, and, IMHO, also in the NAV world in general: to get automated tests in place that enable us to get a quick understanding of the current state of our code with respect to existing functionality (i.e. have regression testing in place). With this I keep thinking, but maybe I am too stubborn in that, that this can best/the most easiest be achieved by keeping close to what we are used to in the NAV world: perform/create functional/integration tests. This is mainly what the MS Test Automation Suite entails and as such is a fairly easy start.
On second thought I agree with your approach. And this is what, in the end, with respect to automated testing, developers should do: write unit tests to their app code. Having said that I realize I still will push my team first to get the integration tests in place.
About #2. “developing for testability” One of the things I’m trying to do in my development work is to use expressions instead of statements as much as possible. I want my functions to be as pure as they can practically be, i.e., fully deterministic and without observable side-effects, in order to optimise their testability.
One of the things I’m trying to do in my development work is to use expressions instead of statements as much as possible. I want my functions to be as pure as they can practically be, i.e., fully deterministic and without observable side-effects, in order to optimise their testability.
Fully agree, even though this is not an easy thing to achieve in NAV due to the habits we have grown in the NAV world. But yes, this makes code easier to test.
About #3. “predefined test data” I’m not sure if I fully understand when you say that your test data baseline should be 100% stable and known, but you are using CRONUS data? We have no real way of knowing what changes Microsoft makes to the demo data between releases, do we? Wouldn’t you be better off generating all of your own data in a new NAV company? Or is it just a trade-off between effort and security?
I’m not sure if I fully understand when you say that your test data baseline should be 100% stable and known, but you are using CRONUS data? We have no real way of knowing what changes Microsoft makes to the demo data between releases, do we? Wouldn’t you be better off generating all of your own data in a new NAV company? Or is it just a trade-off between effort and security?
You're fully right in all aspects. But practically, in our current situation, our daily test run has a 100% identical data baseline as long as we haven't moved to another version of CRONUS. And ... even when we move to a new version ... as long as our tests still prove to be successful, I will call it stable. If the contrary happens I will start considering "generating ... data in a new NAV company".
The rest of your replies make perfect sense to me.
Well, actually, Jan, everything you always wanted to know about automated testing in NAV and did ask me about. ;-)
Thanx for starting this creative little experiment, allowing to elaborate somewhat more on some aspects of automated testing. Maybe needless to mention that my answers reflect my knowledge and experience and are in no way the one-and-only truth. And BTW: my Dutch Dynamics session had a very restricted scope being the way we started using the NAV Test Automation Suite MS is providing us with every release.
So let's go ... I will state your questions
1. First and foremost – how do you decide what (and how) to test? When I first started writing automated tests, I found myself testing things that were so obvious that they probably never posed any risk to the stability of the application in the first place. Can you give some rule of thumbs for what to focus our test effort on?
When I first started writing automated tests, I found myself testing things that were so obvious that they probably never posed any risk to the stability of the application in the first place. Can you give some rule of thumbs for what to focus our test effort on?
Wow, this already is a comprehensive question, as it partly depends on the context. Are you wanting to write tests while your are developing a new feature, or for an already existing one?
Let's approach them apart (even though they still have a lot in common).
To be honest I do not have a standard list on this. But my approach would be (without going into too many details):
If time does not suffice to get this done for each new feature, focus on the business critical ones. Those that:
Now having written this: this actually would be my approach also with ... Existing Features. In the NAV world, we probably all share the same feeling questioning where to start in the humongous sea of features; this heritage of code we built up ever since we started with it. Start to write tests, that will help the most to improve and guard the quality of the code.
2. How does the need for automated testing affect development work? You mentioned that testing NAV (ERP?) is different from testing most other systems, since practically everything goes through the database and there’s no easily available way to mock (simulate) this database interaction. Do the developers in your team have testability in mind when they are writing new features?
You mentioned that testing NAV (ERP?) is different from testing most other systems, since practically everything goes through the database and there’s no easily available way to mock (simulate) this database interaction. Do the developers in your team have testability in mind when they are writing new features?
Well ... I would like to say, yes. But, no, that's unfortunately not the case and this has all to do with where we come from in the NAV world. Developing without:
And yes, as in the NAV world in general, we are improving on this and testability is part of that, but one somewhat more up the road. Now that we started to use the standard Test Automation Suite, testability becomes slowly part of our vocabulary as we do run into parts of our code that we have a hard time getting it tested easily.
3. Using demo data as the basis for your test data You mentioned that tests should ideally create (and clean up) their own data, returning the database to its pristine state after all the tests have run. In our experience, being overly strict about that costs time twice – once during test development, and during each test run. How do you feel about isolating some of the data creation in a demo data creation tool, and running your tests in a database that already has that generated data on board.
You mentioned that tests should ideally create (and clean up) their own data, returning the database to its pristine state after all the tests have run. In our experience, being overly strict about that costs time twice – once during test development, and during each test run. How do you feel about isolating some of the data creation in a demo data creation tool, and running your tests in a database that already has that generated data on board.
Maybe I should mention that the basis to this is that each test should be run from the same baseline to make it reproducible. The baseline in our case is CRONUS, as provided be MS on the product DVD; and what MS also uses as their baseline. Adding your own data that, creates another baseline, so why not? Make sure that in between each test codeunit the state is this baseline.
BTW: an additional reason why we are using CRONUS is that running the Test Automation Suite on a copy of our production database (approx. 600 GB) never ended. It seemed to get stuck on the big number of item ledger entries. However, I never did spend deep investigation in getting to know the exact reason as the CRONUS baseline suffices very well.
4. Have you considered running chunks of tests in parallel? I guess that could significantly reduce the execution time, right? And that becomes even more relevant e.g. when you want to do some form of gated check-in, where tests must pass before a changeset is accepted into your code repository? Also, running in parallel forces you to make your tests fully independent of each other – as they should be.
I guess that could significantly reduce the execution time, right? And that becomes even more relevant e.g. when you want to do some form of gated check-in, where tests must pass before a changeset is accepted into your code repository? Also, running in parallel forces you to make your tests fully independent of each other – as they should be.
Nope. As we're still in the phase of getting all the standard tests working, this hasn't been our focus. But sure we will in due time, like we will also improve they way we are creating our test data now.
5. How do you design new tests? In my experience, designing your tests in a code editor leads to the worst results. I think it’s best to formalise your (existing, manual) tests, i.e. listing the steps and verifications, in a text editor, in plain English before converting them to code. Would you agree?
In my experience, designing your tests in a code editor leads to the worst results. I think it’s best to formalise your (existing, manual) tests, i.e. listing the steps and verifications, in a text editor, in plain English before converting them to code. Would you agree?
Fully agree. No pin between (geen speld tussen te krijgen ;-).
6. Most of our tests were (consciously) implemented as UI tests. Only having access to fields that are visible from the GUI can be quite limiting – there is no straightforward way to get e.g. the Line No. from a Sales Line. Any advice on that (apart from using unit tests instead)?
Only having access to fields that are visible from the GUI can be quite limiting – there is no straightforward way to get e.g. the Line No. from a Sales Line. Any advice on that (apart from using unit tests instead)?
A field like Line No., which, as a design pattern, should no be placed in the GUI, is actually not the only issue. Any field that has Visible=FALSE has a similar issue. For the latter I have asked to allow us to change it's visibility from code. I think something alike should be asked regarding fields like Line No.: to be allowed to access fields not available in the GUI, like you would use the About this Page feature.
7. You mentioned the other day some strange differences between running the test suite from the Windows client, and running it ‘headlessly’ from PowerShell. Can you elaborate a little on that? Did you manage to solve that issue?
Can you elaborate a little on that? Did you manage to solve that issue?
The standard Test Automation Suite has a number of tests (like running reports) that needs a Client session. Running it 'headlessly' from PowerShell yields the error:
Microsoft Dynamics NAV Management Server attempted to issue a client callback ….. Client callbacks are not supported on Microsoft Dynamics NAV Management Server.
Asking MS they confirmed this and also told they have some special tooling to enable 'headless' test runs. The tooling is however not yet ship-able.
Well ... hope this makes sense.
As not only you, Jan, but many others probably have noticed I am doing my best to share as much as possible things regarding the test automation in NAV. Currently it focuses on the Test Automation Suite and you can find a number of NAV Skills webinars I have been performing:
Another one has been scheduled: Testability Framework Deep Dive. Date & Time : July 6th. 10am CET & 4pm EST. (You can find the recording here.)
And during the next NAV TechDays I will lead two pre-conference workshops and co-lead a conference session on automated testing.
So get yourself informed ...!
One of these of every day things that makes work so easy; you just take them for granted. Getting the 2.2.0 update mail from Christian Clausen, I realized that's what Statical Prism has become for me. No fuss of whatever setup, except for having my code in .txt objects in a directory. Using a source repository, that's what is on my system by default. But hey, let's not repeat what I wrote before.
If you haven't used Statical Prism before, go and try it. If you did and stopped because of some missing features, you might want to pick it up again as the 2.2.0 release has some major features added to it:
Just to mention two.
The first was something really missing; I mean, we very often would like to know where the OnInsert trigger was being called, don't we.
The second as a very nice-to-have one. Typically not on my list, but now looking it ... it's neat. In many cases Find Usages on fields yields a long result list. Often overwhelming. Now it's possible to filter this list.
Say you want to find the usages of the primary key (Code) on the Currency table (4). This would be the result:
451 hits, maybe too many just to browse through. Say you're only interested in the usage related to SETRANGE. Use the Text filter to filter out only hits relating SETRANGE, this will show the following:
Once again: great guys!