Bogus Benchmark (was Re: JVM performance on Linux (vs. Solaris/Windows))

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



K P wrote:
Sun's recently published SPECjbb_2005 numbers on Linux, Windows and
Solaris on their
Opteron system, and the Linux result is the lowest of the three by far:

Linux: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.html
Solaris: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00063.html
Windows: http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00064.html

It's not evident if Sun spent any time analyzing and tuning the Linux
result. While the
majority of the tuning opportunities for SPECjbb_2005 are likely to be
in the JVM itself, I was
wondering (given the large spread between the OS's) if there were
other typical opportunities
to tune the Linux kernel for JVM performance and SPECjbb_2005.
----
   I may have missed it, but I didn't see any information on how the
three different binary javaVM's were "built".  What compilers did
they use? What options?  Did they use the OSs' most common compilers
(gcc, MS C, & Sun's C Compiler?).
   There are many other factors affecting performance.  It's
not clear what background processes were running (affecting memory
cache) or if they ran them in single-user mode.  Do the tests
include graphics?  What windowing/graphics software did they
use? Where they running a desktop?
They don't say what kernel version they used (though that might
be implied in the version of RH they used).  They don't
remark on what options were used in mounting the file systems.
They mention that they used "ext3" (which defaults to
"write-through" write operations) vs. NTFS on Windows and Ufs on
Solaris.  NTFS defaults to "write-back" by default.  I'd bet
UFS does as well.
Were the OS and test databases on the same place on disk?  I.e.
loaded on the same partitions?  Read/Write performance varies,
easily by 100% between the slowest and fastest positions on disk.
Was the spec, _at least_, run in it's own partition?

We're talking a difference of 10-15% between the benchmarks.
Any number of factors could influence a benchmark by more than
that: Compiler options, OS & app disk location, background tasks
loaded, file system choice and options.
Speculating that the differences have much of anything to do
with the linux kernel seems awfully premature.

I'd tend to regard this as a fairly worthless benchmark.  In
addition to the questions above, one could still assume they
are using default settings and options.

Anyone concerned with performance is unlikely to be using
defaults.

Linda


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Index of Archives]     [Kernel Newbies]     [Netfilter]     [Bugtraq]     [Photo]     [Stuff]     [Gimp]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Video 4 Linux]     [Linux for the blind]     [Linux Resources]
  Powered by Linux