All Things Distributed

Comments on the benchmark results

I want to briefly address some of the comments that are making their way around the weblogs with respect to the HPC benchmarks postings of this week.

• The research has three main goals:
  1. Collect and/or develop a suite of benchmarks that exercise CLI-based virtual machines from the perspective of high-performance (scientific) computing.
  2. Compare the performance of CLI-based virtual machines with the best JVMs to see whether CLI could be a player in the same field the Java Grande Forum is addressing.
  3. Investigate the performance characteristics in terms of memory consumption, data/instruction cache hits and misses, IO intensity, etc.
• The benchmarks reflect a single moment in time and space. Even though I have made sure that the numbers I am presenting are representative for some set of machine and platforms, your mileage may vary. I have been in the business of large scale systems experimentation long enough to know the difficulties associated with proper benchmark analysis and I have tried to avoid the usual pitfalls with this work as well. You may have a version of a runtime that performs better on your machine, or tomorrow BEA may release a new server VM that will blow the others out of the water.  That is why it is more important to look at what makes these machines perform better, instead of looking at the absolute numbers.
• The benchmarks will be made available this week so you can run them yourself, on your own hardware and your own special platform.
• Mono's performance is improving rapidly. We used mono-0.23 in these tests but as Miguel shows in the comments to yesterday's posting, the current code in the cvs is already faster again. It is not my goal to hammer at the absolute numbers. Mono is improving at a pace that I think their performance will be on par with the other big players soon enough.
• If you are concerned about performance you may want to consider not using SSCLI. It is a great platform for language and runtime research, especially with the excellent book available, but it needs someone to write a new JIT.
• In the micro-benchmarks I am only presenting the numbers from IBM's 1.3.1 JVM. I do this because the focus was on the CLI virtual machines and not the different JVMs. I ran the benchmarks with IBM 1.3.1, BEA Rockit 8.1 and Sun 1.4.1 Hotspot server. On my machines (single and dual Pentium 4 2.8 GHz Xeon, 2 GB memory) and my platform (Windows Server 2003), IBM's JVM performs better than the other two JVMs, so that is why it is used as the bar against which to compare the CLIs. In some of the other results that I will post later this week, I will present all three JVM results.