For that reason, I wanted to share with you three fun and informative stories that were shared with me this week. They are all IT-focused and should give you something to talk about between application performance measurement and development. Enjoy…

First up, an article on the sorry state of IT awareness and security in big companies. It was in InfoWorld and the whole article is here.

Next, another InfoWorld piece on why IT is actually good. See that article here.

And finally, a piece that will get you a little hot and bothered. The story of how financial institutions are cutting back on IT spending. That’s from our friends at NetworkWorld.

As usual, we welcome your news, thoughts, comments and contributions. We’d also like to do video interviews with some of you if you have an application performance story or tip you want to share. Just leave a note in the comments! Thanks.

Software performance engineering is becoming increasingly important to businesses as they look to improve the non-functional performance of applications and get more out of IT investments.  By leveraging performance engineering techniques, IT professionals can be indispensable in building and optimizing scalable systems. This introductory course will teach you the essentials of software performance engineering including :

• The performance challenges faced by Enterprise IT today
• What is software performance engineering (SPE)?
• Best practices for building scalable software systems
• The approaches to integrating SPE into IT project lifecycles
• Common frameworks for measuring application performance and service levels
• The impact of SPE on software developers, testers, capacity planes,
and other IT professionals
• Case studies from the finance, retail, and insurance industries

Instructor: Walter Kuketz, SVP and CTO, Collaborative Consulting

Would love to know what you find out and how you like the session. Please leave your comments here on the blog. Thanks!

While at the Web Performance meetup in Boston last week, I had the pleasure of speaking with William Toll about application performance engineering. We spoke about the importance of prioritizing your goals and paying attention to how well all your systems work together – apps, infrastructure and more. Above is William on camera with a snippet of that conversation.

Also, if you’d like to get on camera and be featured here, leave a comment on this or any post. We’ll either come to you to do an interview and feature your expertise, or we’ll set up a Skype video interview. Thanks!

The increase in virtualization and cloud computing has significantly changed the scope and complexity of capacity management. Forecasting hardware needs is no longer enough. You must understand and optimize your business services, applications, and infrastructure through these new platforms. Join Correlsense and Metron-Athene for this engaging webinar which will explore the latest “need to know” trends in capacity management, including:

• How to more effectively manage your IT costs and SLA agreements
• What you need to know about capacity management when operating in both physical and virtual environments
• How performance monitoring in cloud-based environments relate to your capacity management goals
• What is unique about capacity management and monitoring for virtualized applications
• How to align traditional capacity management techniques with the ITIL methodology

Click on over and sign up. Could be informative and fun.

Performance management of these apps is no less critical when the application is helping a tourist find a restaurant than it is when the application has been written to help efficient IT operations within a multi-national organization. In fact, in many cases, an A.P.E. can make evaluations of speed, performance and availability using consumer apps that are simpler than those crafted and coded in their lab.

We’re talking about a consumer focus today because the 2012 Consumer Electronics Show has just opened in Las Vegas and most of the news is about how BlackBerry and Windows are revitalizing their OS with applications. Further, there’s talk of Android and iOS bolstering their positions with hundreds of new apps that do everything from monitor your home security to give you heads-up displays on the windshields of your car.

With so many players in the consumer market developing applications, those of us on the enterprise side can certainly learn a bit by looking at their engineering path and the hurdles application engineers may have overcome in creating $.99 apps for Joe and Josephine Six-Pack.

Is there a lesson here? I think so. We need to remain aware of the world around us and not operate in a vacuum. Even the most skilled writer can learn something by reading other literary works. And even the most skilled A.P.E. can take away some knowledge by seeing how other IT pros do their work – even if the end product is vastly different in use than are the apps he might produce in-house.

What’s the biggest news you’ve heard on the development front lately? What are you thoughts about the similarities (or differences) in commercial vs. consumer application performance engineering? Leave your thoughts in the comments.

In a recent piece on The Virtualization Practice blog, Bernd Harzog posits the demise of virtualization and the resulting issues that might arise from this situation. For instance, he says that virtualization is moving toward silo’d solutions that become more costly and a burden to manage.

Here’s the complete article with Harzog’s points.

Ultimately, if you don’t have the right tools with which to operate, how can the product you’re creating succeed? To that point, if you’re an application performance specialist, if you can’t measure, monitor and optimize performance, you’re going to be working in the wrong type of walled garden. One that limits your effectiveness and provides you no options for success.

Your take on the demise of virtualization? Leave your thoughts in the comments.

Since I got involved into performance testing in 1997, there were always challenges as how to do performance testing for this product or that technology (see, for example, my old CMG paper: Workload Generation: Does One Approach Fit All?). In my understanding, record/playback is just one performance testing approach – very wide-spread, but definitely not the only one. Although the issues listed by Stephen could be answered even in the record/playback framework.

LoadRunner is a good tool, with the main advantage of supporting numerous protocols. That is the main reason why it became the de-facto standard in the corporate world, where you have hundreds of applications using every possible technology. You don’t see much competition here either – as far as I know, only SilkPerformer is still trying to keep up here.

But as soon as we get to Web only, you get a lot of products to choose from. Keeping in mind LoadRunner’s price, I don’t see why you would pay if you don’t have it already and need Web support only (and especially plain HTTP). There are numerous products on the market to choose from. Although, of course, the more you need, the narrower the list of the products you may use.

Let’s talk about Betfair. it is widely known that the LoadRunner limitation is that it mainly provides analysis after the end of a test. It is not even a record/replay approach issue, it is LoadRunner’s specific issue (and probably some other tools have this issue too). This is one of SOASTA’s main attack points against LoadRunner – that SOASTA CloudTest provides real-time analytics and LoadRunner doesn’t. So Betfair’s approach looks like a workaround for specific LoadRunner features, not a paradigm change.

Next, let’s look at WebSockets. It was not long ago when most products moved from a fat client to a browser-based plain HTML client and scripting was indeed relatively easy. Still, we had Java applets and ActiveX controls, which communicated with the server in their own way. Now we have all kinds of rich Web interfaces (including AJAX, WebSockets, and many other similar technologies), which poses the same kind of challenges. Therefore, I wouldn’t say the current model is broken, but rather that it is constantly changing and needs to be adjusted.

We definitely have new challenges as well, and they may be solved either inside the record/replay approach or outside it. It may be the traditional correlation / parameterization of HTTP scripts,  which still work for many rich Web clients. Or, it may be new protocols – such as the LoadRunner AJAX TruClient which records user interaction with interface objects rather than between client and server. It may be that in some cases we won’t find an available tool and will need to step outside the record / playback paradigm to create our own workload generating test harness.

So, I don’t think that the current model of load/performance testing is broken – but we need to adjust it to whatever challenges we have. These challenges are not only technical as mentioned above, but also existential. It looks like we have load testing somewhat discounted in the Internet companies (I don’t see much load testing discussion at the Velocity conference!)

Performance/load testing is a way to mitigate risks: the risk of being slow, the risk not to function properly under load, the risk of failure. At one point, it was the main way to mitigate these risks. Maybe for some Internet companies the whole risk is not so important – they are not so dependent on customer’s money and downtime doesn’t kill them. A possibility to backup the release quickly is also a way to somewhat to mitigate these risks. Good production monitoring tool (like business transaction monitoring and end-user monitoring) also mitigates these risks. Still, load testing remains the main way to mitigate these risks proactively, giving an opportunity to fix the problem before it impacts business. Finally, it provides the unique way to learn how the system behaves under load, allows for proper system tuning and optimization, and conduct performance troubleshooting in a consistent way.

What are your thoughts on the paradigm of performance/load testing? How have you conducted such measures in your organization?

When requirements are measured as resource utilization, they are related to a particular hardware configuration. There are meaningful metrics when the hardware configuration is known. But these metrics don’t make sense until the hardware configuration is decided upon: how can we talk about processor utilization if we don’t know how many processors we need? Such requirements are not useful for software if it gets deployed to different hardware configurations, and, especially, for Commercial Off-the-Shelf (COTS) software. Furthermore, they don’t work well with new technologies like cloud computing, virtualization, and service-oriented architectures, where hardware resources are shared and /or changing. We speak about resource utilization during early phases of the system lifecycle, but only as a generic policy. For example, a corporate protocol may be that CPU utilization should be below 70 percent.

When required resources are specified in absolute values, it may be considered a performance metric of the software itself, without binding it to a particular hardware configuration. In the mainframe world, MIPS were often used as a metric for CPU consumption, but there is no such widely used metric in the distributed systems world. Metrics like the number of instructions to execute or the number of I/O operations per transaction were sometimes used in modeling, but they were not very practical. Modeling tool vendors probably used some kind of metrics behind the scene, but they didn’t expose them much.

The importance of resource-related requirements is increasing again with the trends of virtualization, cloud computing, and service-oriented architectures. When we depart from the “server(s) per application” model, it becomes difficult to specify requirements as resource utilization, as each application will add only incrementally to resource utilization. There are attempts to introduce absolute metrics, such as the ‘CPU usage in MHz’ or ‘usage MHz’ metric used in the VMware world. The ‘Megacycles’ metric, sometimes used by Microsoft, is another.  Finally, nearly every cloud provider tries to introduce such metrics – Amazon, for example, uses Compute Units.

In the ideal case (when the system is CPU bound and we can scale the system linearly just by adding processors), we could easily find the hardware configuration needed if we had an absolute metric of resources required.

For example, if software needs X units of hardware power per request and a processor has Y units of hardware power, we can calculate the number of such  processors N needed for processing Z requests as N=Z*X/Y. The reality, of course, is more sophisticated. We have different kinds of hardware resources: processors, memory, I/O, and network. Usually we concentrate on the most critical one, keeping in mind others as restrictions.

Nevertheless, it looks like it would be very beneficial to the industry in general to agree on an absolute metric of resources required, and to my surprise, I haven’t heard any discussion about it. There are hundreds of standards produced by IEEE, The Open Group, and other organizations – value of some may be argued about – but it looks like nobody is discussing this missing cornerstone metrics of performance engineering and capacity planning.  Is it the time to agree on one?

The Computer Measurement Group is a not-for-profit, worldwide organization of IT professionals committed to sharing information and best practices focused on ensuring the efficiency and scalability of IT service delivery to the enterprise through measurement, quantitative analysis and forecasting.

CMG members are primarily concerned with performance evaluation of existing systems to maximize performance (eg. response time, throughput, etc.) and with capacity management where planned enhancements to existing systems or the design of new systems are evaluated to find the necessary resources required to provide adequate performance at a reasonable cost

There were many, many very information sessions and panels on Performance and Capacity for all types of systems. A major theme through the conference is the Impact of Cloud computing and performance.

The breadth of the topics is great and they are aimed at all levels (from beginner to advanced). The topics covered; Z/OS performance and capacity, modeling and queuing of systems (three part series within the conference), understanding massive systems and their complexity, the need plan better, nonlinear performance of batch workloads, and many more.

The CMG recognizes excellence in Performance and Capacity, and has a long history with many industry visionaries.  They have a distinguished speaker track and a Michelson award winner track.

There were deep presentations on Performance and VMware, Capacity Management for a cloud deployment and what does capacity mean in the on-demand Cloud?  We heard from many great vendors, one in particular (SOASTA) executed a Cloud Load test with over one million users!!  IBM was present and well represented, discussed the future of Analytics for proactive monitoring of systems and applications.

The crowd came from all over the US and many from Europe, to exchange stories, ideas and lessons learned from working in the field.

I recommend using the CMG as a source of performance practices and guidelines and information.

Where you in attendance at the National CMG meeting? Have you attended a CMG meeting before? What are your thoughts on this group?

This year, the secret sauce of speed and performance went to Apple and JC Penney. They came out on top in a cyber Monday test that pitted multiple commercial Websites against each other in site speed tests. What this meant – according to the folks at Information Week – is that companies were better prepared to take orders and ultimately boost their bottom line during this first wave of holiday shopping.

The article I read says there was 19% growth in sales this year and that can be loosely attributed to the belief that shoppers who get to their products and checkout faster are more likely to complete their orders. Frustrated shoppers tend to be more likely to leave sites that don’t perform well.

Is there a lesson here? Not particularly, because any A.P.E. worth his salt is already focused on making products, applications and Websites as efficient and available as possible. Perhaps the fact we should all take away is that the competition to be the best is not letting up. Last year’s winner, Dell, was knocked down a few spots to end up in fourth this year. So, the need for attention to detail and performance engineering skills is not going away…and it’s likely going to continue to be a vital component for all online and traditional businesses moving forward.

What do you think about the rankings? Is this a true measure of A.P.E. results or are there better ways to evaluate it?