Network Performance Testing

At work we use the Anue Systems Ethernet Network Emulator for testing various parts of our products.  Most typically, we use them for emulating a WAN between various products – they allow us to precisely simulate network conditions including loss, latency, jitter, and many other properties.  They’re also useful when you want to see how a web interface deals with these sorts of scenarios.

Note: in the pictures included, it appears that the Anue systems are made by Spirent – this is not the case.  Anue had a reseller agreement with Spirent at the time we purchased the equipment, and it came branded as Spirent, but we still deal with Anue for support and follow on work.

The Anue has been rock-solid and very reliable.  It’s integrated into our automation via their TCL interface, and we step through many different scenarios this way.  We also gather a wealth of information from the Anue to back up the statistics we are also gathering from the network test harness, and the product under test.  Overall it’s been one of the least problematic pieces of test gear we’ve ever used.

It boots super fast, and has one of the most to-the-point and speedy web UI’s.  No client software required, and everything is very clearly labeled and easy to read.   I like graphical UI’s to do initial setup when learning, but not when using frequently.  Many test vendors provide drag and drop functionality which I find impedes general test creation when you’re doing things frequently.

One of the nicest things I like about their web UI is when you make a change to a value, the value and field around it turn yellow, so you know you need to submit your changes before things will become active.  This is an excellent visual cue that things aren’t done yet.

The level of accuracy of the Anue is amazing.  We’ve compared it against other solutions and Anue wins hands down.  This undoubtedly has to do with their FPGA hardware architecture, and is also why they can maintain line rate without any issues.  The configurability and features are matched with many of the other vendors, and in many areas much easier to use in Anue.  They have the ability to change just about anything in the packets flowing through them, if you want.

We also use the Anue as a router within our test harnesses in some cases.  It has the ability to emulate router IP addresses and act as a default gateway.  This functionality was added in the last few years to their product, and was a requirement for our testing to simplify the amount of equipment involved.  Previously we had been using an actual router for this functionality.

There are a few things that are slightly annoying.  First, and most intrusive, a full reboot clears your entire configuration other than the management IP.  Nice if you’ve screwed things up, but not so nice if you didn’t intentionally reboot.  You can back up your configuration, or use scripts to set things up, but it would be nice to have more control over this – i.e. make it a switch whether things are erased.

One of the other things, which is an oversight in their API, is the ability to configure router IPs remotely via TCL.  This is worsened by the above problem.  The typical scenario is that when you reboot, or if the box reboots due to a power blip or otherwise, you need to connect to the GUI, re configure the router IPs, and then push the rest of your config up to the box via the TCL automation scripts.  Thankfully, the system only reboots infrequently, and there’s hope that configuring the routing functionality will be exposed to the TCL API sometime soon.

Overall, the Anue is a solid product with an easy to use GUI, reliable and detailed API, and highly accurate feature set.  As far as WAN emulators go, after having tested and used the major hardware and many smaller software vendors, this is the one I would use in any deployment because of reliability and ease of use hands down.   I like it when things work well, and deliver consistent reliable operation!

Have been thinking a lot about testing WAN optimization devices lately, and am surprised at how many of the test scenarios required to validate a product like this don’t exist in the market yet.

One of the missing pieces is control over the randomization of content.  Many vendors offer the ability to generate random content, but in some cases the overall file size is extremely limited (typically no larger than 10MB) or the “randomness” is not really random.    I’d like to see the randomness be controlled by dynamic percentages – varying it so that you can emulate the various scenarios required for testing compression related algorithms.

The other missing piece is the size of files capable of being retrieved from a server – this is an issue that has existed with almost everyone’s solution for years, and although it’s getting more attention as of late because of this and other markets, is still not fully addressed.  Some vendors have limits of less than 1MB, while others can serve some files out as large as 100MB but either with severe performance limitations, or other restrictions.

File size limitations are really frustrating – this should have been addressed ages ago.  Typical downloads of files, even for things like Power Point or Word documents in a corporate environment where WAN optimzation is present, are often times several megabytes.   Outside of WAN optimization testing, typical downloads on the internet are far larger.

It’s understandable based on many test vendors architectures that putting files of any serious size, or randomized files, is very challenging, or in some cases, impossible.  Many of the architectures have an extremely limited space to store files because they are optimized for connections per second or other more CPU intensive items.   However, from the pure file size perspective, all of these architectures could adapt to grab a file from a network location and serve it out – almost always the limitation here isn’t CPU cycles on the test gear, but bandwidth or throughput.  Obviously I don’t know all the intracacies of everyone’s architecture, but it seems like just larger files would be an easy challenge to solve.

Randomized content would definitely be more difficult if you were doing it on the fly.  Making things completely random all of the time, or even a percentage of it, could be CPU intensive to the test equipment, but I bet there are some options there, whether they be hardware offloaded or not.

There are some good resources out there for WAN optimization testing, including the Silesia Corpus, which includes a library of files of varying compressibility and content types, like medical X-rays, text files, images, etc.  Even integrating something like this into the test gear would be great.    Of course it doesn’t solve the need for large datasets of random content for testing memory and disk caches.

To solve this now, I’ve built real servers that house large random datasets.  This is the approach that others I’ve talked to have done as well.  The problem here is that the server isn’t instrumented for stats like a traditional performance harness, so you have a whole set of additional pain and problem points, not to mention potential test failures that you won’t be aware of.  Additionally, it requires a decent amount of coordination work on the client side to make sure you’re actually pulling the right objects in your test.

Many vendors have expressed interest in this market and are working on solutions.  At least until there are solutions from some of the big ones, this leaves a gap for the smaller or newer players to fill, and I’d say rather easily given some of their architectures.  The WAN optimization market is growing very well even in these tough economic times, so there’s money to be had pretty easily.  Hopefully we’ll be seeing some new solutions here soon!

For a while I have been contemplating opening the forums back up again, as it seems there are more folks out there both reading the blog, and interested in commenting on things.  I’ve had a bunch of folks emailing me on various fronts, and also a few people posting comments on the blog.

I don’t think I would give up the blog – I would integrate it somehow into the forums, but I’m wondering if there is enough interest to have an open forum not affiliated with a test vendor, and allow for articles, a link directory, and blog space for anyone who registers.

If you’re interested, please post a comment or drop me an email…

Spirent announced a few weeks ago a new solution, TestCenter Virtual, for validating the performance of all elements of the data center and cloud computing environments including virtual machines (VM), servers and storage devices.

In examining the details, it appears that TestCenter Virtual is a module that is installed in each VM which will allow for more granular control of the overall test by actually generating traffic from the VM’s themselves.

The product has been selected as a finalist for Best of Interop 2009.

I think it’s a great idea, along the lines of IxVM announced earlier this month by IXIA.  Allowing purpose-built performance test equipment to control virtual machines will allow for folks to validate all of the virtual infrastructure, and possibly also be a great way to scale testing without having to purchase more performance equipment.

I am very interested to see these products once they are out and available to the general public.

IXIA announced a test, which they will be showing at Interop this week, of Arista Networks’ new 48-port 10GBASE-T solution.  They used their IxYukon 10GBASE-T solution, which can be scaled up to 96 ports per chassis.

From a pure deployment scenario, I haven’t been seeing a lot of folks using 10GBASE-T yet, but it is an interesting choice.  In thinking about fiber versus copper, most of the solutions I’ve seen have been focused on fiber, and most people have built the infrastructure to support that.  I can see 10GBASE-T being popular within an individual rack itself, as it is much easier to cable, and not as prone to problems with damage, etc.

However, most of the articles I’ve read focus on folks re-using CAT5e/6 architecture for 10GBASE-T which is also an interesting idea.  I think we’ll see this pick up over the next year or two as folks add more and more virtual computing clusters that require larger data connections, but I’m not sure how many people actually use it now…

Anyhow, interesting test, and definitely an interesting switch from Arista.  For those of you at Interop, you should stop by IXIA’s booth and check it out.  I’m not there this year, so it’d be great if someone can post here any of their thoughts on the test itself at Interop.

I’ve been looking forward to evaluating BreakingPoint’s Elite system for some time.  Over a year ago, I had the chance to see their previous system, the 10K, but at the time it couldn’t test the device that my company makes, so we moved on.

We have the Elite in house now, and so far it’s interesting.  So far the evaluation is going well – I’m on my own much of this week (by design) to fumble around and find the different things I’m interested in.  So far I have successful connection per second and throughput tests going through our device under test, and have fiddled with a bunch of the options to tune things exactly the way I want.

There’s a lot left to figure out – perhaps I’ll post some of my thoughts later this week after we’re done.

The second in a series of webinars by TesLA is happening this week.  I’m excited because I’ll be speaking on the challenges with automating test architectures, and how I believe TesLA will help solve these challenges.  In particular, I’m looking forward in discussing interoperability standards and the challenges now with automating different devices all in one test architecture.

Stop by and join us this Thursday at 10AM PST – link to webinar below.

Hope to see you there!

Webinar Details:

Thurs., May 14 @ 10 a.m. PST – 2nd Installment of TesLA Webinar Series, Steve Mitchell of F5 (TesLA CAB member)
Register ahead of time at: https://www2.gotomeeting.com/register/886587307

IXIA announced today their plans to perform a big test at Interop with Force10 for 10GE switching.  It will require 280 IxYukon 10GE test ports, and end up sending and receiving 5.6 terabits per second of traffic.  Quite a bit of traffic!

Seems that  lot of people are doing the bigger/faster/more test stuff – Spirent did a 10GB test back in October that was, at the time, the largest.  BreakingPoint also did a big test with Juniper back in March, although that test was markedly different in the goals and type of test.

Those of you going to Interop (I’m not) should stop by and let us know what it looks like.

Check out a conversation I had recently with Kyle Flaherty over at BreakingPoint on the evolution of performance testers….

IXIA announced today another successful test of 100GBPS technologies, this time via their Higher Speed Ethernet load module, known as K2, and using a  CFP MSA  compatible optical module.  They’re going to have it on display during Interop – hopefully they’ll be more info then too.

10 months ago we saw another demo from IXIA, which I wrote about.  At the time, we were using forums instead of the blog, and there were a lot of comments from some sharp folks, all mostly centering around the fact that a truly agreed-upon standard for 100GBPS technologies hasn’t been hammered out yet.  Things have changed a bit since then, but there’s still some contention.

It’s good to see vendors like IXIA getting things working this early in the process.  I’m hopeful that the IEEE will approve things on time (in 2010?) for Higher Speed Ethernet, and we’ll be able to use technologies like the K2 for production ready products then.

Info on Higher Speed Ethernet:

• IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force
• IEEE 802.3ba on Wikipedia