Where Are the iSeries Benchmark Tests?

Originally Published: June 2, 2003, at IT Jungle

by Timothy Prickett Morgan

Note: the important bit in this story is shown in red below. This the first moment I started thinking about open source servers and benchmarks for them, which are aslo when IT Jungle was getting ready to turn on its first internal data center.

You would think that with a totally revamped and repriced iSeries product line, IBM would be eager to demonstrate the power and value of the machines running on various benchmark workloads commonly used in the IT industry. But, alas, IBM has not been very active with iSeries benchmarks, and now the lack of good benchmark data is hampering the ability of the iSeries platform to gain new customers and to retain the ones it has.

Back when the TPC-C online transaction processing benchmark test was announced, in the early 1990s, IBM's E series AS/400s were the first machines tested. And IBM tested not only a high-end, four-way E95, but also small and midsized machines, to demonstrate the scalability and pricing at various points across the AS/400 product line. Coupled with the RAMP-C relative performance metrics that IBM supplied for each AS/400 server--a predecessor to the CPW ratings IBM gives for each iSeries processor today--customers could work out where any AS/400 machine stood on the TPC-C test. It's been a long time since IBM ran the TPC-C test on an entry or midrange AS/400 or iSeries server, and it has been years since it has run tests on entry pSeries Unix boxes. The last iSeries TPC-C benchmark test was run on an iSeries Model 840 back in December 2000.

I know that tests like the TPC-C benchmark are expensive to do. But the fact is that all of the Unix vendors and the iSeries team look like they have something to hide when they don't run benchmarks. This is particularly true at the low end, where the onslaught of inexpensive Wintel and Lintel machines has made it extremely difficult for these vendors to show equivalent performance, price/performance, or both. The iSeries has advantages that the TPC-C test does not demonstrate--like the ability to run mixed workloads really well on a single machine--and maybe it is time for IBM to lead the way again, as it did with the TPC-A, TPC-B, and TPC-C tests, and to create a new and better benchmark that is easier and cheaper to run. What the industry needs is an inexpensive test with audited results and list pricing that actually shows how companies really use machines like the iSeries. I'll be the first to agree that the TPC-C test, the best test there is, is not really doing this.

In the absence of actual benchmarks, we are left to make estimates. I did this in March and April, first comparing the new iSeries to Unix servers and then comparing the revamped line to Windows boxes. My comparisons were intended to give server buyers a sense of where these machines might stand against each other; they were never intended to actually replace audited benchmark tests. Only benchmarks can show where the oddities in server architecture and pricing pop up. This creates a feedback loop, whereby vendors then improve their systems and tweak their pricing to better match competition. This results in happy customers.

Let me give you a small example from last week. IBM cut prices on the 64 GB main memory cards used in the high-end 16-way pSeries 670 and 32-way pSeries 690 Unix servers by 25 percent. This price cut is not just aimed at helping customers spend less on main memory. This cut in particular is designed to make the pSeries 690 "Regatta-H" server using 1.7 GHz Power4+ processors beat the forthcoming Windows-based Superdome server, from Hewlett-Packard, using 1.5 GHz Itanium 2 processors on the TPC-C online transaction processing benchmark test. HP has just leap-frogged IBM's best results on the Regatta-H server, and it is working on a new test to jump ahead of HP's latest result with that Windows-Itanium Superdome.

Anyone who tells you that benchmarks don't drive the pace of innovation and price cuts in the server business is either a liar or a fool. I just think it is unfortunate that there is not a consumer protection law compelling all server vendors to run a suite of benchmarks on all of their machines. I may start lobbying for such a thing if server vendors don't start behaving themselves. The alternative, of course, is to run our own benchmarks, through an open-source benchmark consortium of user companies that buy and use all of these machines. We don't need vendors to do tests. We just need access to machines and a desire to know their performance. I like this idea better, because it is grassroots and doesn't involve lawyers or politicians.

Giants Slugging It Out Above Our Heads

HP became king of the hill again on the TPC-C benchmark last week with that 64-way Itanium-based Superdome server. The Superdome, running the new Windows Server 2003 Datacenter Edition from Microsoft, has bested IBM's 32-way pSeries 690 using the new Power4+ processors. How long HP will hold the top spot is unclear, with IBM said to be working on its own revised benchmark that will put it atop HP's latest result.

The two Superdome servers that HP tested had exactly the same configuration and price. They were configured with 64 of the 1.5 GHz/6 MB cache "Madison" processors, expected to ship this summer from Intel. The Superdomes had 512 GB of main memory. The central electronics complex of this Superdome cost around $2.2 million, the main memory another $3.3 million, and $2 million of the core server coming from the 34.6 TB of disk storage and related peripherals. The 64-bit versions of the Windows 2003 Server Datacenter Edition operating system and SQL Server 2000 Enterprise Edition database cost $1.3 million on these Superdomes. Including application servers and three years of maintenance, the Superdomes cost $10.5 million. But then HP invoked a Large Systems Discount of 38.5 percent, and dropped the price tag of these servers to $6.5 million.

To make the debut of Windows Server 2003 on April 24, the best that the techies at HP's performance labs could do was 658,278 transactions per minute (TPM) on the TPC-C online transaction processing benchmark, which simulates order-entry and supply-chain applications. The resulting price/performance of the first benchmark run on the Superdome Windows setup yielded a bang for the buck of $9.80, which is the first time that an enterprise server in the RISC/Unix class of machine (even though it is Itanium-Windows box, this is clearly what it is) has gotten below $10 per TPM. On the second run, announced today, HP was able to push 707,102 TPM, an increase of 7 percent over the first pass. That increased throughput, entirely due to tuning, dropped the price/performance of the Superdome Windows setup to $9.13 per TPM. Why these machines will not be available until October 23 is unclear. Microsoft says that all implementations (32-bit Pentium or 64-bit Itanium) and editions of Windows Server 2003 are shipping.

Incidentally, sources at HP that I spoke to last week say that they can do even better in terms of throughput on a Superdome Itanium server running its own HP-UX Unix operating system, which also runs on the same machine. Exactly when HP is going to prove this is unclear, but the company may be keeping some of its powder dry for the aftermath of IBM's next Regatta benchmark, which is expected to be configured running Oracle9i and which is probably going to hit somewhere in the range of 720,000 TPM with essentially the same hardware configuration as the DB2 Universal Database setup IBM just announced two weeks ago. HP is expected to ship HP-UX Version 2 for Itanium at the end of July, and this may be when it announces such benchmarks. This timing for the new HP-UX is probably not a coincidence and is probably when the Madison chip will launch.

That IBM Regatta-H server was configured with AIX 5L V5.2 and IBM's DB2 Universal Database 8.1 software. The server had 32 1.7 GHz Power4+ processor cores (with a 1.5 MB shared, on-chip L2 cache for every two cores), 512 MB of L3 cache, 512 GB of main memory, and 43 TB of IBM's 7103 SSA disk arrays. That Regatta setup server could do 680,613 TPM. The pSeries 690 server cost $3.27 million, with $1.38 million going for main memory alone and $1.4 million going for processors. The SSA disk subsystems cost $6.86 million, and AIX and DB2 for the server cost $632,725. With application servers and three years of maintenance thrown in, the whole TPC-C configuration had a list price of $14.57 million, but after a 48 percent discount, the price of the whole shebang dropped to $7.57 million. IBM was able to show a price/performance of $11.13 per TPM after the discounts. This configuration will be available November 8. Why it is not available right now is unclear.

No matter when the updated Regatta server is available, IBM is obviously counting on the fact that on real-world workloads a lower processor count is going to make Regatta appealing against Superdome regardless of the spread on the TPC-C benchmark. Processor count is important for application pricing and on single-threaded jobs, like batch applications, the speed of a single processor is often as important than aggregate OLTP throughput in a box. That said, IBM is probably shooting for under $10 per TPM on the next Regatta TPC-C benchmark, just to give HP trouble.

Here's some iSeries perspective. By my estimates, the fastest iSeries Model 890 server, a 32-way machine using 1.3 GHz Power4 processors, can probably hit around 350,000 TPM on the TPC-C test, maybe 400,000 with some database tuning and large memory support (like 512 GB of main memory, which is not yet supported on the box). The same 1.3 GHz machine with a pSeries 690 label running AIX and Oracle was tested a few months ago and came in at just under 430,000 TPM with that half-terabyte of main memory recently. Heaven only knows what the cost of such a configuration might be, but given the new iSeries pricing, it should not be that far off the pSeries list pricing for similar equipment. However, it seems highly unlikely that IBM will put a 48 percent discount on an iSeries configuration to make it look competitive with Unix boxes. So the public number would probably be in the range of $20 per TPM, and that is not a very useful message to give to prospective buyers. Now you know why IBM hasn't published a TPC-C benchmark test on the iSeries Model 890, even though it probably has run it in its labs.