In my post on calculating throughput and response time I discussed how to measure the average throughput and response time for a given amount of time. I think it’s important to understand how to calculate these averages, but providing a single average causes you to mask other important trends and information about the system behavior.
This is a given in any statistical analysis. We don’t want to publish a large set of data because there is too much information, so we try to extract important information. But if we shrink the derived data set too much then it is hard to use this information to extrapolate any other trends in the data.
This is why I think it is valuable to capture all the data you can, derive averages, and create graphs.
It is valuable to capture and keep as much data as possible from a test run. This allows you to go back and derive new trends, averages, and graphs. I like to capture a message count for each second of a test run. Let’s say I’m sending messages into an asynchronous application and receiving messages out of the system after they have been processed. A table of this type of data might look like this:
This table gives me a lot of information. For time 0 seconds to time 1 second, I sent 5 transactions into my system and received no messages out of my system. From time 3 seconds to 4 seconds, I sent 10 messages into my system and received 5 messages out of my system… and so on.
In order to convert this raw data into something more useful, I may derive some averages. Providing averages is important because it allows someone to quickly compare one set of data to another. If I run a test script 10 times, and 1 run has an average throughput that is 40% lower than all the others, then I immediately know that something went wrong. But of course, it is hard to extract anything further from a single number.
For the sample data above, I might extract the following averages:
Input rate = M / Ts = 92 messages / (11 - 0) seconds = 8.36 messages / second
Output rate = M / Tr = 92 messages / (12 - 2) seconds = 9.2 messages / second
Throughput = M / Tp = 92 messages / (12 - 0) seconds = 7.67 messages / second
Tss = Send Start Time
Tse = Send End Time
Trs = Receive Start Time
Tre = Receive End Time
Ts = Send Time = Tse - Tss
Tr = Receive Time = Tre - Trs
Tp = Processing Time = Tre - Tss
M = # of messages
Note: Yes, the output rate for a system can be greater than the input rate to a system. Think of a line to a roller coaster ride. People can trickle in to line for a ride at a fairly slow rate. But when the gates open, people quickly get out of line to board the roller coaster. The output rate is greater than the input rate.
Graphs are also important because it is easier to visualize trends. Graphs are also pretty and can score bonus points with managers and business-types. I think it is useful to used stacked graphs that show the behavior of one trend compared to another. For the data above, we may create a stacked graph:
This link is very useful if you want to create stacked charts with vertical separation in Excel. I used this as a guide in creating the graph you see above.
It is equally useful to create graphs of your system response time as a function of time. This can show you how and when your system starts to degrade.