Automating with Test Conductor
I’ve spent a lot of time in the last few days working with IXIA’s Test Conductor product, which is the cornerstone to our Performance Harness Project. I’ve used it in the past, and know the basic portions of it pretty well.
This time around, we’re focusing on two main use cases for this harness:
- Quick baseline tests between builds of product during the development process.
- Complex scenario testing that can be automated
For the first one, Test Conductor has a lot of great features that we’ve used in the past to help. The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline. It’s really quite slick and has saved us a lot of time – first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values. Then, you run your regression and tell it to collect baseline values. Once that is done, you are ready to run against something and see if the performance has varied. Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result.
It’s a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse – manual examination. And it doesn’t have what many other solutions have – hard coded values and engineer input. In our use case, we simply reset the baseline on the next build after we’ve validated things are OK, and get ready for the next build coming down the line.
We’ve designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible. It’s not the kitchen sink, just a quick validation point. This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems.
The second use case is one we’re hoping to take more advantage of with Test Conductor – complex scenarios that can be automated. Why do I say “can be automated”? Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it’s worth. We test many of these already and have great solutions for them. What I’m looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with.
So far I’ve come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be. In particular the functionality in Test Composer is going to be key to making all of this happen. So far we’ve created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression – all in just a day or two. And, without coding in TCL or some other language. Key features we’re using so far include DUT Configurations, Procedures, Response Maps, and more. As we get specific examples tested and better documented, we’ll post a couple here.