Oracle compares Apples with Oranges in recent SAP benchmark

IBM’er Elisabeth Stahl talks about what to make of Oracle’s recent SAP Assemble-to-Order Standard Application Benchmark.

————————————————-

What’s in a benchmark, Oracle?

I spent most of this past weekend looking at hardware … hardware for furniture, that is. I was in search of drawer-pull hardware, and when it comes to this, I like old. I looked at silver, satin nickel, flat black, even something called “brushed English gold,” but nothing could beat the old antique look of oil rubbed bronze.

With technology, of course, that’s a different story. But you would never know that with some of the news going around lately.

Oracle recently published an M9000 with 11g result on the SAP Assemble-to-Order (ATO) Standard Application Benchmark (1). When I first read the press release, I thought this result surely must be impressive.  After all, it is “record-breaking,” “highest score ever posted,” and “beats all.” Then I tried to remember the last time I had seen a result on this benchmark. When was the last time I had heard anyone even talking about this benchmark? I checked the SAP site and saw that Fujitsu had published a couple of results, the most recent being in 2003.

The only thing more dated in technology this week than Oracle’s benchmark is HP’s Anniversary Calculator.

Oracle published another press release, claiming a second “world-record” SAP benchmark result. Let’s understand what’s behind this one.

Oracle compares SAP SD Parallel benchmark data with SAP SD standard data. Note that these are different benchmarks, and have very different data distributions. As highlighted on the SAP website for these benchmarks, “it is generally accepted that data distribution can significantly influence the benchmark result.”

Like “Larry” and “Leonard,” the benchmarks clearly have different names:  SD vs. SD-Parallel. This is for a reason. The benchmarks are located on different SAP web page listings, also for a reason.

IBM has the #1 result on the very popular and well-recognized SAP benchmark, the SD 2-tier benchmark. (See footnote #2)

Even if we were to agree to be ignorant and compare an SD benchmark result with an SD-Parallel benchmark result, does using 87 percent more cores (as Oracle does) to get six percent more users sound right to you?  Think about it. That would result in overpaying for a system that get six percent more users.

************************************************

(1)Two-tier SAP ATO standard application benchmark (SPARC Enterprise M9000 Server, 64 processors, 3.0 GHz, 256 cores and 512 threads running Oracle Database 11g Release 2 and Oracle Solaris 10) 206,360 fully business processed assembly orders per hour. Certification number 2011033.  Source: www.sap.com/benchmark (Results are current as of September 6, 2011).

(2) Two-tier SAP SD standard application benchmark: 126,063 SAP SD benchmark users, 0.98 seconds average dialog response time, 13,772,670 fully processed order line items per hour, 41,318,000 dialog steps per hour, 688,630 SAPS. Average database request time (dialog/update): 0.011 sec / 0.024 sec. CPU utilization of central server: 96 percent. Server configuration: IBM Power 795, 32 processors / 256 cores / 1024 threads, POWER7, 4.00 GHz, 32 KB (I) and 32 KB (D) L1 cache and 256 KB L2 cache per core, 4 MB L3 cache per core, 4096 GB main memory gain running AIX 7.1 operating system, DB2 9.7 and SAP enhancement package 4 for SAP ERP 6.0. Certification number: 2010046 vs. Two-tier SAP SD-Parallel: 134,080 SAP SD-Parallel benchmark users, 0.87 seconds average dialog response time, 14,805,000 fully processed order line items per hour, 44,415,000 dialog steps per hour, 740,250 SAPS, 0.011 seconds/0.048 seconds average dialog response time (dialog/update). CPU utilization of servers, 92% (node 1 active: 89%, node 2 active: 93%, node 3 active: 93%, node 4 active: 93%, node 5 active: 92%, node 6 active: 92%). Server configuration: 6 Sun Fire X4800 servers each with 8 processors / 80 cores / 160 threads, Intel Xeon Processor E7-8870, 2.40 GHz, 64 KB L1 cache and 256 KB L2 cache per core, 30 MB L3 cache per processor, 512 GB main memory, Solaris 10 operating system, Oracle Database 11g Release 2 with Oracle Real Application Clusters, SAP enhancement package 4 for SAP ERP 6.0. Certification Number 2011036.  Source: www.sap.com/benchmark (results current as of September 13, 2011).

SAP, mySAP and other SAP product and service names mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world.  Oracle and Java are registered trademarks of Oracle and/or its affiliates.