Network Performance Testing

Have been thinking a lot about testing WAN optimization devices lately, and am surprised at how many of the test scenarios required to validate a product like this don’t exist in the market yet.

One of the missing pieces is control over the randomization of content.  Many vendors offer the ability to generate random content, but in some cases the overall file size is extremely limited (typically no larger than 10MB) or the “randomness” is not really random.    I’d like to see the randomness be controlled by dynamic percentages – varying it so that you can emulate the various scenarios required for testing compression related algorithms.

The other missing piece is the size of files capable of being retrieved from a server – this is an issue that has existed with almost everyone’s solution for years, and although it’s getting more attention as of late because of this and other markets, is still not fully addressed.  Some vendors have limits of less than 1MB, while others can serve some files out as large as 100MB but either with severe performance limitations, or other restrictions.

File size limitations are really frustrating – this should have been addressed ages ago.  Typical downloads of files, even for things like Power Point or Word documents in a corporate environment where WAN optimzation is present, are often times several megabytes.   Outside of WAN optimization testing, typical downloads on the internet are far larger.

It’s understandable based on many test vendors architectures that putting files of any serious size, or randomized files, is very challenging, or in some cases, impossible.  Many of the architectures have an extremely limited space to store files because they are optimized for connections per second or other more CPU intensive items.   However, from the pure file size perspective, all of these architectures could adapt to grab a file from a network location and serve it out – almost always the limitation here isn’t CPU cycles on the test gear, but bandwidth or throughput.  Obviously I don’t know all the intracacies of everyone’s architecture, but it seems like just larger files would be an easy challenge to solve.

Randomized content would definitely be more difficult if you were doing it on the fly.  Making things completely random all of the time, or even a percentage of it, could be CPU intensive to the test equipment, but I bet there are some options there, whether they be hardware offloaded or not.

There are some good resources out there for WAN optimization testing, including the Silesia Corpus, which includes a library of files of varying compressibility and content types, like medical X-rays, text files, images, etc.  Even integrating something like this into the test gear would be great.    Of course it doesn’t solve the need for large datasets of random content for testing memory and disk caches.

To solve this now, I’ve built real servers that house large random datasets.  This is the approach that others I’ve talked to have done as well.  The problem here is that the server isn’t instrumented for stats like a traditional performance harness, so you have a whole set of additional pain and problem points, not to mention potential test failures that you won’t be aware of.  Additionally, it requires a decent amount of coordination work on the client side to make sure you’re actually pulling the right objects in your test.

Many vendors have expressed interest in this market and are working on solutions.  At least until there are solutions from some of the big ones, this leaves a gap for the smaller or newer players to fill, and I’d say rather easily given some of their architectures.  The WAN optimization market is growing very well even in these tough economic times, so there’s money to be had pretty easily.  Hopefully we’ll be seeing some new solutions here soon!

Check out a conversation I had recently with Kyle Flaherty over at BreakingPoint on the evolution of performance testers….

I thought I’d share a link to a document that the company I work for produced a bit over a year ago entitled Creating a Robust Performance Testing Methodology.

Although some parts of it are specific to Application Delivery Controllers, the general set of steps and ideas behind the process are excellent.

Yes, I’m sure I’m a bit biased, but even trying to set that aside, every time I am stuck trying to figure out what the best approach is, this document always has a little tidbit that can help me along the way.

I wish there was some sort of standardized methodology outline that folks could all use when performance testing network devices – perhaps that’s a good thread to start here…

One of the biggest reasons I see performance testing fail is the non scientific approach to testing that usually ensues when something strange happens. Most of the time it causes a cascade effect in the test cycle – further tests get more crazy until the result you’re looking at makes no sense, and trying to connect how you got to the last point is impossible.

Unfortunately, doing the right thing takes a lot more time. It requires a lot of patience and control to not jump ahead, and usually a lot of data collection.

Here are some tips I’ve figured out over time that help when I test performance. You may notice that it is similar to what a scientist might record when doing an experiment (albeit much simpler) – that’s the analogy I use when explaining why performance testing is a difficult job, and time consuming.

1. Outline - come up with a general outline of where you think your testing will go. It doesn’t have to be exhaustive, but it will help you figure out how to approach things. It will be really difficult to do #4 below (one variable change at a time!) if you don’t have a clear plan as to what you’re trying to prove/find/fix and the areas of the product/device you’re suspecting things will be a problem.

2. Write it down – accurate notes of what you are testing, how things were configured, and the trail is critical in figuring out which result is the most useful. Try to use the test tool if possible to make notes about each individual test. Be as verbose as possible. My rule of thumb is that you should spend at least 5 minutes after each test noting the results in some format that will allow you to understand everything you tested a year later when you need to recall the information.

Saving the test data is critical (see below) but your observations and notes are even more important so you can remember why things happened the way they did, and what you observed.

3. Keep all your data – if your test tool supports it, every time you make a change, change the name of the test, or note the change you made. If it doesn’t support it, figure out a way to combine tip #2 above with this so that you can accurately reproduce the test later.

I go even further and have a script that prompts me for my notes, gathers the configuration from the devices I’m testing with, and the device I’m testing, and package it all up in a simple tarball for use in the future. Someone could take this entire bundle and completely reproduce what I did, along with reading my general notes on the test.

4. One variable at a time! – this is probably the most important tip of all – only change one variable at a time – anywhere in your test environment. Many people will try this for a while, and when the result doesn’t look good, or as the steps above get mundane, multiple things are changed, and then the trail can’t be reproduced.

Often I have talked with folks about not changing too many things, and they aren’t able to do it because of time restrictions. If that is the case, there are ways to optimize the above (like a script) that can make it still workable to take it a step at a time.

One of the best ways to make all of the above happen is to be in complete control of all of your equipment. Much of the testing I’ve seen that fails is when there are too many cooks in the kitchen – one person running the test gear, one person watching the device under test, and each of them watching differing things, and possibly making changes that don’t match.

I’m sure there are a lot of other tips that folks could suggest, but these are the four biggest ones I’ve tried to stick to always, with good results.

Good luck!