There are many questions which must be answered to determine what software performance requirements exist for both business stakeholders and engineers. This post will articulate a step-by-step methodology for successfully determining requirements.
This first question that must be asked is: “what type of application is it?” The performance goals must be negotiated between the business stakeholders and the application Architect. To have a productive and two-way negotiation, you must understand the type of application and the amount of data manipulated for each transaction. For instance, generating a large report covering sales for that last two quarters, will take longer than added a product to an order. And you must have supporting data, from current production systems. You must have measurements from the current system on the key business transactions.
• Online web transactions, e-commerce, browser based and mobile based. These transactions are typically synchronous with a customer or business partner waiting for the responses.
• Message based transactions: computer to computer transactions, what is the distance between the systems and what is the latency of the network. Typically, they are part of a larger workflow.
• Business intelligence and reporting with on-demand reports
• Data capture or collection from sensors
• Calculation engine: Pricing, medical claim adjudication, insurance policy rating
• Are there specific penalties regarding stability and availability, and performance?
• Are there stringent time based deadlines that must be enforced?
1. Have your historical data ready for the business discussion, be able to tell them how much it costs to achieve each level of performance; sub-second vs. two second response times.
2. Understand how the application is categorized and how that influences the performance goals. The industry you are in will make a big difference in the performance goals.
Critical business transactions
First, you and the business must understand the user response time satisfaction index; there are four levels of the index, satisfied, tolerating, frustrated and abandonment. Some transactions will be fine with 3 second response time, other will need 1 second.
Then, what are the critical business transactions?
• Define the response time for a satisfied user experience for those key business transactions.
• Define the response time for a frustrated user experience for those key business transactions.
For message-based systems; what are the throughput requirements? How many prices must be calculated per second, how many claims must be adjudicated per second?
Here are some basic questions that every architect/developer/ performance engineer must be able to answer for the application:
• Who uses your application?
• How many people use the application?
• What do they use it for?
• Why do they use it?
• When do they use your application?
• From where do they access your application?
Define the conditions that these performance goals need to operate in. This is the workload discussion. What is the average number of users, and what is the peak number of concurrent users?
If you have 500,000 registered users, how many access the application?
What is the business’ tolerance for failure? At what point does a degraded transaction response time become unacceptable and hurtful to the business? A transaction, such as order status, may have a response time of two seconds under normal load. How does is slow down under heavy load? It may go to 4/6/8 seconds. It may just not respond. The architect must decide when to turn users away due to load. You want to make sure that users in the application continue to process under load and not allow new users into to the system until they can be processed safely.
You must have a scalability plan defined for peak load conditions.
Take-away: requirements phase
1) List all the key transactions and be specific with the transactions and the satisfied response times. For instance, the Order Status transaction is satisfied at 2 seconds for the 90th percentile under normal load. Under peak load, tolerated response time is 3 seconds for the 90th percentile.
2) Have a clearly defined scalability plan for flash events and for normal business growth for online transactions and calculation engines. For instance, the adjudication engine under normal load must process 20 claims per second. At month end peak, it must process 40 claims per second.
3) Communicate the performance goals clearly to the next phase of your SDLC. The architect team must be fully aware of these performance and scalability requirements. The goals will be validated in the performance testing process.