Network Performance Testing

I’ve spent a lot of time in the last few days working with IXIA’s Test Conductor product, which is the cornerstone to our Performance Harness Project.  I’ve used it in the past, and know the basic portions of it pretty well.

This time around, we’re focusing on two main use cases for this harness:

• Quick baseline tests between builds of product during the development process.
• Complex scenario testing that can be automated

For the first one, Test Conductor has a lot of great features that we’ve used in the past to help.  The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline.  It’s really quite slick and has saved us a lot of time – first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values.  Then, you run your regression and tell it to collect baseline values.  Once that is done, you are ready to run against something and see if the performance has varied.  Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result.

It’s a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse – manual examination.  And it doesn’t have what many other solutions have – hard coded values and engineer input.  In our use case, we simply reset the baseline on the next build after we’ve validated things are OK, and get ready for the next build coming down the line.

We’ve designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible.  It’s not the kitchen sink, just a quick validation point.  This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems.

The second use case is one we’re hoping to take more advantage of with Test Conductor – complex scenarios that can be automated.  Why do I say “can be automated”?   Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it’s worth.  We test many of these already and have great solutions for them.  What I’m looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with.

So far I’ve come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be.  In particular the functionality in Test Composer is going to be key to making all of this happen.  So far we’ve created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression – all in just a day or two.  And, without coding in TCL or some other language.  Key features we’re using so far include DUT Configurations, Procedures, Response Maps, and more.  As we get specific examples tested and better documented, we’ll post a couple here.

Shenick announced today that they are partnering with Fanfare to add full plug in support to the Shenick diversifEye™ product.  According to the press release, this will allow users of Fanfare’s iTest product to control and manipulate per-flow testing from the Shenick product.

A lot of the press release is focused on Service Providers and wireless markets, which is an area that Shenick focuses on more than many of the other players.  Direct from the press release:

“NGN and LTE services and applications bring with them increased quality of service (QoS) and quality of experience (QoE) demands. Per-flow test and measurement is the only mechanism to guarantee performance at these granular levels enabling providers to pinpoint application and individual user issues accurately”

I think it’s great that this support has been added on both ends so that users of iTest can take advantage of existing or new Shenick equipment.  I hope there will be more details about how deep the integration goes in the coming weeks.

One interesting thing is the fact that both Fanfare and Shenick are part of the TesLA Alliance – I wonder how much the proposed TesLA standards have influenced this solution, and if we’ll see more of this in the coming months since the standards are getting closer to being approved…

I’m in the process of setting up a new set of test equipment that includes a lot of different vendors, pieces and parts.  My goal is to have a completely automated harness that tests various aspects of performance and is able to reconfigure devices, harnesses, switches, and all other pieces from one simple central console.  I’m not writing any code, using any TCL or other languages – my goal is to use Test Conductor from IXIA to drive the majority of this, but interface with a bunch of other vendors equipment.  So far, here’s the list:

• IXIA Test Conductor – central orchestrator, test and regression management
• IXIA X16 chassis with 8 ALM cards – generates L4-L7 traffic using the IxLoad application
• IXIA 400T chassis with 1 TXS card – generates L2-L3 traffic using the IxAutomate application
• ANUE GEM Network Emulator – used to emulate WAN conditions
• APCON IntellaPatch L1 switch – for changing port configurations and connecting/disconnecting devices
• Dell PowerEdge 1950 server - running CentOS 5.3 with a special Chelsio CX4 10GB network card and tuned software – more on this later
• Dell PowerConnect 6248 stacked switches – three of these provide all of the switching I need
• F5 BIG-IP 3600s – the devices I’m testing

So far that’s my equipment list.  I have a few other items I will likely be adding as the days go on, and may be able to scrounge up some additional interesting scenarios.

I will be writing here about the progress I’ve made with each device and test scenario, and discussing a bit about each device and the things I’ve found to be interesting.  Off we go!

The second in a series of webinars by TesLA is happening this week.  I’m excited because I’ll be speaking on the challenges with automating test architectures, and how I believe TesLA will help solve these challenges.  In particular, I’m looking forward in discussing interoperability standards and the challenges now with automating different devices all in one test architecture.

Stop by and join us this Thursday at 10AM PST – link to webinar below.

Hope to see you there!

Webinar Details:

Thurs., May 14 @ 10 a.m. PST – 2nd Installment of TesLA Webinar Series, Steve Mitchell of F5 (TesLA CAB member)
Register ahead of time at: https://www2.gotomeeting.com/register/886587307

Not a lot of new news out there in the last few weeks.  Have been looking around diligently for something new and exciting, but it seems folks are hunkering down for a while.

At work, we’re deep into figuring out lab and test bed automation, and reviewing a bunch of solutions from various vendors.  I hope to have more details for a posting in a week or two…

I’ve been looking around lately for lab automation solutions – something that has an open API of some sort that can be called by all of the other automation solutions we use at work to provision things like L1/L2 switches, configure DUTs and devices in the flow of traffic, and provide scheduling abilities for all of those devices.

There are a few solutions out there, like one from Gale Technologies called Lab Manager.  Some others that are a little less well known, and hard to find information on.  I was hoping I could find some sort of open source engine that handled device scheduling and contention, and that we could adapt to provide our configuration and control for the devices we have in our lab.  So far no luck.

Has anyone else found other Lab Automation solutions out there?  I’d love to start a page for links for this sort of stuff since it seems hard to find…

I’m on the Customer Advisory Board for the TesLA Alliance and we had our second quarterly meeting this last week.  I missed the first meeting due to being sick, so it was great to finally meet the rest of the TesLA Members and the Customer Advisory Board as well.

I am really excited about all of the new things that we’re planning to do, and very appreciative to be involved with the entire alliance.  The caliber of all of the members and board is top-notch, and there are many very exciting dicsussions going on that will forever change this industry in the positive.

Why is it so exciting to me?

Well, first you have a collection of Members that span the performance, functional, and lab automation arenas.  Some of these vendors are bitter rivals, yet they met together, and all agreed on many things that will help to not only unify the market, but provide far more benefits to end customers.  There are even more vendors that are interested, and attended the first part of the meeting to see what it was all about.

Second, you have the goals of the alliance, which are exactly what I’ve been looking for.  Defined and published standards.  Framework to allow vendors to communicate between each other.  Completely open to all vendors.  An alliance focused on all of these things, and not on building their own product.

Third, the momentum behind this is amazing, and the amount of progress that has been made in the last few months shows that this is being done at the right time, and the right place.  So many people agree that the industry needs this ASAP.

Finally, the customer advisory board – my colleagues are all top-drawer folks from varying parts of the industry.  One of the members was adding up the experience we all have together and we came up with something around 80 years!  We’re the reality check for the rest of the alliance to make sure that what they’re coming up with meets customer expectations.  It’s a privilege to work with these folks on such a critical standard for our industry.

What’s next?

I would like to see even more vendors participate in the alliance.  There are some noticeable key players that are still playing politics, when they should really just put those aside and get on board.

There’s a lot more planned, and a lot of new things happening every week.  Check out the Activities Calendar, or subscribe to the RSS feed for the site.  I’m looking forward to the ongoing participation, as well as the new folks who will be joining.

Does anyone know what’s happened with Gale Technologies?  Their news page has been empty for months, and the only other updates are a few customer stories…

Many of you will remember that Gale was formed by the merger of QuikCycle and Edentree which I wrote about back in October.  Both were pretty major players in lab and test automation.   I sure hope we’ll see something new from them soon!

UPDATE: They are very much still alive – I bumped into some of them a few days ago at the TesLA Alliance Quarterly Meeting.

Some really good news today – the TesLA Alliance announced the first draft of a test automation standard.  I’m really excited to see this at the TesLA quarterly meeting coming up next week, which I will be attending.

TesLA is also hosting their first of a series of webinars starting this week.  The first one is entitled “Open Testing Standards Emerge: Enabling Seamless, Shareable and Platform-Independent Automation” and features HP ProCurve’s Vince Mendoza who is part of the TesLA Customer Advisory Board.