What Really Matters in TPC-C Benchmark Results?
Oracle currently claim to have the “Fastest Ever Database Performance”. They make this claim based on their leading result for the TPC-C industry benchmark. IBMer Conor O’Mahony asks the Smarter Question: “What really matters in a TPC-C benchmark result?” Over to Conor…
Love it or hate it, TPC-C is the industry’s leading transactional benchmark. Many deride the TPC benchmarks, including Forrester who claim that TPC Benchmarks Don’t Matter Anymore. However, others consider them to be useful. For instance, Merv Adrian blogs about Database Benchmarks – The Gift That Keeps on Giving.
Expensive benchmarks aren’t highly relevant to your organization
Forrester make valid points in their report. The TPC-C benchmark has indeed turned into an arms race, with the vendor that is willing to spend the most on a benchmark system taking the lead. IBM and Oracle have been swapping lead changes for some time, and sometimes in spectacular fashion. Are these huge benchmark systems applicable to your environment? Probably not. Very few organizations need to process anything even remotely approaching these volumes of transactions (and those that do, are often secretive). The likelihood is that your organization needs to process onlya tiny fraction of the number of transactions in one of these benchmark systems.
If these massive benchmark systems have little direct relevance for the majority of organizations out there, and if the raw throughput of these massive systems is becoming less relevant, what should people focus on?
Efficiency is a better measure database performance
We should focus on both processing efficiency and cost efficiency. Processing efficiency is not currently a TPC-C metric, although if you do a little investigation and math, you can usually come up with a reasonable analysis. For instance, here is A Closer Examination of Oracle’s “Database Performance” Advertisement where I look at tpmC per CPU core, which focuses on the number of transactions per CPU rather than simply the total number of transactions without context.Whereas cost efficiency is actually reported by the TPC Price/Performance metric (although you need to be aware of Sun and Oracle TPC Price/Performance Tactics, because IBM and Oracle use different tactics for TPC/C Price/Performance numbers). By focusing on the efficiency of the systems for the benchmark results, we hone in on what is important… not how many transactions a system can process, but how efficiently it process those transactions.