Network Performance Testing

Shenick announced today that they are partnering with Fanfare to add full plug in support to the Shenick diversifEye™ product.  According to the press release, this will allow users of Fanfare’s iTest product to control and manipulate per-flow testing from the Shenick product.

A lot of the press release is focused on Service Providers and wireless markets, which is an area that Shenick focuses on more than many of the other players.  Direct from the press release:

“NGN and LTE services and applications bring with them increased quality of service (QoS) and quality of experience (QoE) demands. Per-flow test and measurement is the only mechanism to guarantee performance at these granular levels enabling providers to pinpoint application and individual user issues accurately”

I think it’s great that this support has been added on both ends so that users of iTest can take advantage of existing or new Shenick equipment.  I hope there will be more details about how deep the integration goes in the coming weeks.

One interesting thing is the fact that both Fanfare and Shenick are part of the TesLA Alliance – I wonder how much the proposed TesLA standards have influenced this solution, and if we’ll see more of this in the coming months since the standards are getting closer to being approved…

I’m in the process of setting up a new set of test equipment that includes a lot of different vendors, pieces and parts.  My goal is to have a completely automated harness that tests various aspects of performance and is able to reconfigure devices, harnesses, switches, and all other pieces from one simple central console.  I’m not writing any code, using any TCL or other languages – my goal is to use Test Conductor from IXIA to drive the majority of this, but interface with a bunch of other vendors equipment.  So far, here’s the list:

• IXIA Test Conductor – central orchestrator, test and regression management
• IXIA X16 chassis with 8 ALM cards – generates L4-L7 traffic using the IxLoad application
• IXIA 400T chassis with 1 TXS card – generates L2-L3 traffic using the IxAutomate application
• ANUE GEM Network Emulator – used to emulate WAN conditions
• APCON IntellaPatch L1 switch – for changing port configurations and connecting/disconnecting devices
• Dell PowerEdge 1950 server - running CentOS 5.3 with a special Chelsio CX4 10GB network card and tuned software – more on this later
• Dell PowerConnect 6248 stacked switches – three of these provide all of the switching I need
• F5 BIG-IP 3600s – the devices I’m testing

So far that’s my equipment list.  I have a few other items I will likely be adding as the days go on, and may be able to scrounge up some additional interesting scenarios.

I will be writing here about the progress I’ve made with each device and test scenario, and discussing a bit about each device and the things I’ve found to be interesting.  Off we go!

Back in 2005, we had a bake-off for switch vendors at the company where I work to pick a large port count, line rate switch that included 1GB and 10GB ports.  Since that time, we’ve gotten a lot of good mileage out of the switches we chose, which were Foundry RX-series chassis.  We ended up with one RX-16 and one RX-8 switch, all fully loaded with power supplies, switch fabric, and management modules.

We have a small amount of 24 port 1GB copper cards – the majority of the cards are the RX-Bi48T which is a 48 port MRJ-21 connector style card.  It requires a lot of additional cabling and breakout boxes, but it’s worth it to get more out of your switch.

The Foundry architecture for the 1GB 24 port cards only allows 20 of the 24 ports to operate at line rate, but we’ve planned our architecture not to be a problem.  The 48 port cards only allow 40 of the 48 ports.  Again, we’ve planned around this limitation.

In addition to the cards above, we  use the 4 port 10GB card for our larger devices, and as interconnects between the two RX’es, and many smaller switches such as the Force10 S50 and Foundry FESX-448.

We are moving away from Foundry equipment, however, due to a lot of failures over the last year and the need for a more port-dense 10GB switch for our larger products.  The Force10 C300 is a very attractive option – it’s significantly less expensive that the RX, and many of the other switches out there, and performs better in many cases – line rate in the case of the C300.  We have several already in-house and have been very impressed.

In all cases, the moment we get a new switch, we do many L2/L3 tests with various IXIA and Spirent devices to make sure we understand the limitations.  In the last few years, we haven’t found any issues with the new vendors and switches we’ve come across – at least none of the serious line rate issues we found in the past, including with the RX-series chassis.

Looking forward to spending more time with Force10 products in the near future.

At work we use the Anue Systems Ethernet Network Emulator for testing various parts of our products.  Most typically, we use them for emulating a WAN between various products – they allow us to precisely simulate network conditions including loss, latency, jitter, and many other properties.  They’re also useful when you want to see how a web interface deals with these sorts of scenarios.

Note: in the pictures included, it appears that the Anue systems are made by Spirent – this is not the case.  Anue had a reseller agreement with Spirent at the time we purchased the equipment, and it came branded as Spirent, but we still deal with Anue for support and follow on work.

The Anue has been rock-solid and very reliable.  It’s integrated into our automation via their TCL interface, and we step through many different scenarios this way.  We also gather a wealth of information from the Anue to back up the statistics we are also gathering from the network test harness, and the product under test.  Overall it’s been one of the least problematic pieces of test gear we’ve ever used.

It boots super fast, and has one of the most to-the-point and speedy web UI’s.  No client software required, and everything is very clearly labeled and easy to read.   I like graphical UI’s to do initial setup when learning, but not when using frequently.  Many test vendors provide drag and drop functionality which I find impedes general test creation when you’re doing things frequently.

One of the nicest things I like about their web UI is when you make a change to a value, the value and field around it turn yellow, so you know you need to submit your changes before things will become active.  This is an excellent visual cue that things aren’t done yet.

The level of accuracy of the Anue is amazing.  We’ve compared it against other solutions and Anue wins hands down.  This undoubtedly has to do with their FPGA hardware architecture, and is also why they can maintain line rate without any issues.  The configurability and features are matched with many of the other vendors, and in many areas much easier to use in Anue.  They have the ability to change just about anything in the packets flowing through them, if you want.

We also use the Anue as a router within our test harnesses in some cases.  It has the ability to emulate router IP addresses and act as a default gateway.  This functionality was added in the last few years to their product, and was a requirement for our testing to simplify the amount of equipment involved.  Previously we had been using an actual router for this functionality.

There are a few things that are slightly annoying.  First, and most intrusive, a full reboot clears your entire configuration other than the management IP.  Nice if you’ve screwed things up, but not so nice if you didn’t intentionally reboot.  You can back up your configuration, or use scripts to set things up, but it would be nice to have more control over this – i.e. make it a switch whether things are erased.

One of the other things, which is an oversight in their API, is the ability to configure router IPs remotely via TCL.  This is worsened by the above problem.  The typical scenario is that when you reboot, or if the box reboots due to a power blip or otherwise, you need to connect to the GUI, re configure the router IPs, and then push the rest of your config up to the box via the TCL automation scripts.  Thankfully, the system only reboots infrequently, and there’s hope that configuring the routing functionality will be exposed to the TCL API sometime soon.

Overall, the Anue is a solid product with an easy to use GUI, reliable and detailed API, and highly accurate feature set.  As far as WAN emulators go, after having tested and used the major hardware and many smaller software vendors, this is the one I would use in any deployment because of reliability and ease of use hands down.   I like it when things work well, and deliver consistent reliable operation!

Have been thinking a lot about testing WAN optimization devices lately, and am surprised at how many of the test scenarios required to validate a product like this don’t exist in the market yet.

One of the missing pieces is control over the randomization of content.  Many vendors offer the ability to generate random content, but in some cases the overall file size is extremely limited (typically no larger than 10MB) or the “randomness” is not really random.    I’d like to see the randomness be controlled by dynamic percentages – varying it so that you can emulate the various scenarios required for testing compression related algorithms.

The other missing piece is the size of files capable of being retrieved from a server – this is an issue that has existed with almost everyone’s solution for years, and although it’s getting more attention as of late because of this and other markets, is still not fully addressed.  Some vendors have limits of less than 1MB, while others can serve some files out as large as 100MB but either with severe performance limitations, or other restrictions.

File size limitations are really frustrating – this should have been addressed ages ago.  Typical downloads of files, even for things like Power Point or Word documents in a corporate environment where WAN optimzation is present, are often times several megabytes.   Outside of WAN optimization testing, typical downloads on the internet are far larger.

It’s understandable based on many test vendors architectures that putting files of any serious size, or randomized files, is very challenging, or in some cases, impossible.  Many of the architectures have an extremely limited space to store files because they are optimized for connections per second or other more CPU intensive items.   However, from the pure file size perspective, all of these architectures could adapt to grab a file from a network location and serve it out – almost always the limitation here isn’t CPU cycles on the test gear, but bandwidth or throughput.  Obviously I don’t know all the intracacies of everyone’s architecture, but it seems like just larger files would be an easy challenge to solve.

Randomized content would definitely be more difficult if you were doing it on the fly.  Making things completely random all of the time, or even a percentage of it, could be CPU intensive to the test equipment, but I bet there are some options there, whether they be hardware offloaded or not.

There are some good resources out there for WAN optimization testing, including the Silesia Corpus, which includes a library of files of varying compressibility and content types, like medical X-rays, text files, images, etc.  Even integrating something like this into the test gear would be great.    Of course it doesn’t solve the need for large datasets of random content for testing memory and disk caches.

To solve this now, I’ve built real servers that house large random datasets.  This is the approach that others I’ve talked to have done as well.  The problem here is that the server isn’t instrumented for stats like a traditional performance harness, so you have a whole set of additional pain and problem points, not to mention potential test failures that you won’t be aware of.  Additionally, it requires a decent amount of coordination work on the client side to make sure you’re actually pulling the right objects in your test.

Many vendors have expressed interest in this market and are working on solutions.  At least until there are solutions from some of the big ones, this leaves a gap for the smaller or newer players to fill, and I’d say rather easily given some of their architectures.  The WAN optimization market is growing very well even in these tough economic times, so there’s money to be had pretty easily.  Hopefully we’ll be seeing some new solutions here soon!