Network Performance Testing

Some of you have emailed asking what has happened to the Performance Harness Project I started writing about a few months ago.  I wanted to give you all an update…

The project started as something I was spearheading myself, and has since taken on a life of it’s own.  It’s now an official project with larger involvement, which means that it slows down a bit   I hope to have some more details in early January as we flesh out the new architecture we’ll be using for the system.

Developers on my team have spent the last month prototyping a new reservation and scheduling system that will allow us to control more devices from a central point within our automation.  This will include things like L1/L2/L3 switches, our own product, and various test harnesses from IXIA, Spirent, ANUE, and others.  Part of this framework will be used in this new harness.   What we foresee right now is a mixed harness – automated build testing during the day using our customized framework, and more complex customer scenarios during the night, using IXIA Test Conductor.

We have set aside two brand-new IXIA Acceleron-NP cards for this project – a huge amount of test capacity (more than 7 times what we originally had dedicated to the project) and re-tooled some of the ANUE and Spirent equipment involved.  We also may be adding some larger 10GB switches to support the different devices we’re planning on testing.

More details in a few weeks!

The lack of posts on the new performance harness is due to a few other projects taking priority.  I’m back on track now, and expect some additional posts shortly!

I’ve spent a lot of time in the last few days working with IXIA’s Test Conductor product, which is the cornerstone to our Performance Harness Project.  I’ve used it in the past, and know the basic portions of it pretty well.

This time around, we’re focusing on two main use cases for this harness:

• Quick baseline tests between builds of product during the development process.
• Complex scenario testing that can be automated

For the first one, Test Conductor has a lot of great features that we’ve used in the past to help.  The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline.  It’s really quite slick and has saved us a lot of time – first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values.  Then, you run your regression and tell it to collect baseline values.  Once that is done, you are ready to run against something and see if the performance has varied.  Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result.

It’s a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse – manual examination.  And it doesn’t have what many other solutions have – hard coded values and engineer input.  In our use case, we simply reset the baseline on the next build after we’ve validated things are OK, and get ready for the next build coming down the line.

We’ve designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible.  It’s not the kitchen sink, just a quick validation point.  This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems.

The second use case is one we’re hoping to take more advantage of with Test Conductor – complex scenarios that can be automated.  Why do I say “can be automated”?   Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it’s worth.  We test many of these already and have great solutions for them.  What I’m looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with.

So far I’ve come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be.  In particular the functionality in Test Composer is going to be key to making all of this happen.  So far we’ve created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression – all in just a day or two.  And, without coding in TCL or some other language.  Key features we’re using so far include DUT Configurations, Procedures, Response Maps, and more.  As we get specific examples tested and better documented, we’ll post a couple here.

Performance Harness Diagram

Part of my documentation is diagram of all of the equipment.  Some of this will include physical connections for those items that are static, but there will be a large portion of the equipment that is dynamic and changing based on the use of the Apcon Intellapatch switch we have in the mix.

In diagramming this stuff, I’ve found, to no big surprise, that vendors still don’t keep up with Visio templates.  What I was surprised about is how many vendors don’t keep up!

IXIA doesn’t have any of their newer chassis in the templates they make available through a 3rd party.  ANUE doesn’t have any templates I can find at all (hence the orange box in my diagram to the left).

Of those that had templates that were semi current, APCON’s came with fully populated chassis instead of the individual shapes so you could add the specific cards you want.  I’m sure you could unlock the shape, deconstruct some of it, and then go from there.  They do include a number of different cards in the template, just not undocked from the chassis.  F5 has some nice updated templates, but they seem more marketing than anything – 3D shadows and all.

Dell was the only vendor to have accurate and well thought out templates with the right shapes, sizes, combos, views (both front and back for everything) along with just about everything else you could think of.

Not that I use Visio constantly, and want pretty shapes, but I would have thought this would be pretty straightforward for vendors to keep updated.  Oh well.  Back to diagram-land…

Today I spent some time documenting the performance harness I’m building.  Not that exciting – I’m really more interested in building tests and experimenting with automation and such, but it’s a necessary evil.  So far I’ve found the following places that I have to document:

• Apcon ports and connections
• Dell switch ports
• F5 BIG-IP ports and cabling
• All versions of software/firmware

So far, the majority of this documentation is within the systems themselves.  For Apcon, this was a little difficult to find in the web UI, but once there, it took a few minutes to input all of the port names.

The Dell switch is a known quantity as we’ve used these many times before for other projects.  That doesn’t make it any easier, as the syntax is slightly off from any of the others out there.  For instance, port names are unnecessarily long, and there are extraneous words you have to use to switch to a specific ethernet interface.  They’ve also chosen some really weird abbreviations for the 10GB ports – x – which is really hard to remember.  Regardless of their unique config, they are very good performing switches.  There was a Tolly evaluation of the Dell PowerConnect 6248 a while ago that really pushed us towards using these for many projects since they are so well performing, and for not a lot of money.

I’ve also started creating a high level network diagram which I hope to post here shortly to show how everything is connected.  Of course, the hope is that the Apcon and Test Conductor will allow me to make this a constantly changing network setup based on the various topology goals I have in mind, so the diagram will just be a starting point.

Finally, I’ve created a single link to see all the articles on the harness, which can be found here.  More info tomorrow…

I’m in the process of setting up a new set of test equipment that includes a lot of different vendors, pieces and parts.  My goal is to have a completely automated harness that tests various aspects of performance and is able to reconfigure devices, harnesses, switches, and all other pieces from one simple central console.  I’m not writing any code, using any TCL or other languages – my goal is to use Test Conductor from IXIA to drive the majority of this, but interface with a bunch of other vendors equipment.  So far, here’s the list:

• IXIA Test Conductor – central orchestrator, test and regression management
• IXIA X16 chassis with 8 ALM cards – generates L4-L7 traffic using the IxLoad application
• IXIA 400T chassis with 1 TXS card – generates L2-L3 traffic using the IxAutomate application
• ANUE GEM Network Emulator – used to emulate WAN conditions
• APCON IntellaPatch L1 switch – for changing port configurations and connecting/disconnecting devices
• Dell PowerEdge 1950 server - running CentOS 5.3 with a special Chelsio CX4 10GB network card and tuned software – more on this later
• Dell PowerConnect 6248 stacked switches – three of these provide all of the switching I need
• F5 BIG-IP 3600s – the devices I’m testing

So far that’s my equipment list.  I have a few other items I will likely be adding as the days go on, and may be able to scrounge up some additional interesting scenarios.

I will be writing here about the progress I’ve made with each device and test scenario, and discussing a bit about each device and the things I’ve found to be interesting.  Off we go!