I was asked to outlline what I’d expect someone to decide before they started a performance test exercise. A quick look at StickyMinds shows this document, which looks very good. It is, however, 8 pages long, which is about 7.5 pages longer than I need if my target audience is going to bother with it. Here, then, is my simplified version. Slightly cryptic, perhaps, but workable (at least for a web server, which is what we’re dealing with).
Purpose?
*Performance Test – see what it will do
*Stress Test – see the most it will do
*Load Test – see what happens when it does more than that
*Sizing – Guide to customers on hardware requirements
Test platform?
*Number of machines
*Machine specs
*Network spec
Test Application?
*Low complexity to concentrate on platform
*High complexity to simulate real-life loads
*Think times emphasize load or real-life use.
Finally, how will results be measured?
*Throughput (or equivalents)
*Constraints (e.g. is app under test memory bound, CPU bound, network bound)
*Recommended hardware – e.g. X users per Y GB of memory and Z GHz of processor.
5 Comments
It is an unnecessarily long doc. However I like it because it covers the all-important “burstiness test”.
It’s good it calls that out – too many people confuse burstiness with spikiness, and we all know what happens then…
Both bursty and spikey, that’s very thorough and fits nicely with the boilerplate verbosity
In all seriousness: approach it a bit like a scientific paper, that doesn’t mean you have to write in a dry manner though. Assume you are writing in order that a peer could reproduce your results. I’d add:
- How you will produce the load?
- What will you measure in order to determine the constraints?
- What could mitigate the constraints, what are the trade-offs?
Sizing, that’s a slippery slope when a customer’s workload could be vastly different from your application. But you already know that
Phil – excellent points, especially the first one. I wrote this up in part to prompt the person to realize that he couldn’t just ‘do’ a bit of performance testing, so the mention of sizing was there as much to scare him as anything
A former employer sent me to an MIT-run performance boot camp a few years ago where we learned how to do performance optimization. The most important thing we learned was how to make an M/M/1 model of the system you want to test, which would allow you to literally clock a single path through the code (i.e. a single request) and arrive at how many of those requests you could do per second with the given hardware using a calculator. It was so accurate that a lot of other testing can be forgone if you start here.
You still have to test how your app will react when it’s “overloaded” and some other tests, too. But you can save a lot of time with M/M/1 modeling because you’ll do less of them.