Challenging Oracle’s SPARC SuperCluster Claims

Conor O’Mahony, an IBM employee, closely watches Oracle. In this post, Conor examines Oracle’s recent advertisement related to its SPARC SuperCluster product.

This week, Oracle placed the following advertisement in the Wall Street Journal:

This advertisement claims to compare Oracle’s SPARC SuperCluster with IBM’s fastest computer. The first thing that comes to mind is that the IBM Power Systems 795 is not IBM’s fastest computer. The zEnterprise mainframe system is IBM’s fastest computer. But let’s move past this point, and address the specific comparison presented here.

Lack of Transparency

The advertisement purports to compare the SPARC SuperCluster T4-4 with IBM Power Systems 795. It claims that the SuperCluster is twice as fast as the Power Systems server for Java and Oracle. In the online footnotes for this advertisement, it claims the performance comparison is based on Oracle internal testing. That’s not a lot of information to go on.

The only public comparison between the new SPARC SuperCluster and any IBM Power Systems server for Java performance is Oracle’s recently published SPECjEnterprise2010 benchmark result. I suspect that Oracle are using these benchmark systems as the basis for the claims in the advertisement, and that they are extrapolating IBM Power Systems 795 performance from the IBM Power Systems 780 benchmark result (using published performance factors between the systems).

Let’s analyze this benchmark data, as it is the only available published data.

Oracle is 1.77x More Expensive than IBM

In their product announcements this week, Oracle have been careful to restrict their SPECjEnterprise2010 price/performance comparisons to the application server layer of the systems. They choose to omit the database layer from their comparisons. As you read on, its going to become apparent why they did this. Let’s look at the list price (including one year of maintenance) for the servers, storage, and software needed for the two benchmark systems in question. I think its fair to compare list prices, as discount levels are so variable.

Oracle Configuration
   $1,334,788    4x SPARC T4-4 & WebLogic & system software
   $3,636,521    2x SPARC T4-4 & Oracle Database & system software
   $1,573,470    Storage
   $6,544,779    Total Oracle list price, including 1 year of maintenance

IBM Configuration
   $1,656,747    IBM Power 780 & WebSphere & system software
   $1,651,448    IBM Power 750 & DB2 & system software
   $  395,080    Storage
   $3,703,275    Total IBM list price, including 1 year of maintenance

 
The first thing you’ll notice is the large investment by Oracle in the database layer of their benchmark system. You will also notice the significantly higher spending on storage by Oracle. When you look at the total list price for these SPECjEnterprise2010 benchmark systems, you can see that the list price for the Oracle configuration is 1.77x times the list price of the IBM configuration. It’s interesting how the analysis changes considerably when you take the entire system into account.

Note: I did not include the networking aspects of the benchmark systems in this analysis because I haven’t been able to track down the exact pricing for the switches in the IBM configuration yet. The 1-year costs for acquisition and maintenance of switches in the Oracle configuration is approximately $329k. I expect the 1-year costs for acquisition and maintenance of switches in the IBM configuration to be approximately $20k. I’ll add an update to the bottom of this post when I get this information. But be aware that the price/performance analysis below is in reality better for IBM than presented below.

Oracle has 1.36x the Price/Performance of IBM

In the advertisement, Oracle are claiming 8x better price performance. Perhaps that is true if they:

1. Exclude the parts of their system that cost a lot (like database software and storage)
2. Take the data from one IBM system and extrapolate it to a more expensive system, to further skew the results in their favor

However, if you look at total list price for the servers, storage, and software involved in these benchmark systems, and then figure out the price/performance, it works out that Oracle is 1.36x times the price/performance of IBM.

Oh, and here’s an interesting twist. Correct me if I’m wrong, but I believe that Oracle didn’t actually use a SPARC SuperCluster for this benchmark result. The SPARC SuperCluster comes with the zFS storage appliance, whereas this benchmark instead use F5100 flash arrays (which clearly provide better performance).

Take it with a Grain of Salt

I can imagine Oracle responding to this analysis by saying “thank you IBM for proving that Oracle is better.” However, the only thing I believe I have proven is that you should be suspicious of Oracle claims. When a supposed 8x advantage transpires to possibly be a 1.36x advantage, it has to make you wonder.

Also, you need to take any conclusions based on this benchmark data with a grain of salt. On the one hand, we know that benchmark results do not necessarily translate into real world production results. And we must keep in mind that benchmarks are like a game of leapfrog, with vendors continually surpassing one another. It is very likely that, with the next SPECjEnterprise2010 benchmark result from IBM, that the tables will turn and IBM will have a better price/performance. And so, the dance will continue…

Welcoming an Oracle Response…

Perhaps, I am mistaken in my suspicion that Oracle have based their claims on this benchmark data, and perhaps they have performed their own private closed testing to arrive at these claims. In either case, I would encourage Oracle to publish substantiation for their claims so we can get to the bottom of this. Did they exclude certain parts of the system from their analysis? Did they extrapolate both the performance and price from one IBM system to another? Are they really comparing with their SPARC SuperCluster, or a system that it similar to it? In the mean time, I believe I have presented an open, transparent, and reasonable counter-claim that is based on industry benchmark results and complete data.

This post expresses Conor’s opinions and do not necessarily represent IBM’s positions or opinions. You can read more from Conor on his Database Diary blog.