This quote came from a piece of email today asking me about why we, as in the Drizzle project, have not been publishing comparison benchmarks.
First of all, this is not a Sun request, this was a request from me to the other core members of the project.
Let me explain...
I don't think projects can ever objectively compare themselves to other projects. Look at all of the heat that went on for years between Postgres and MySQL. I think we, the Drizzle Project, are better off publishing code and helping others create benchmarks, but avoiding doing the comparisons ourselves.
We are inherently biased.
One of the first things that went up on the Drizzle wiki were the pages that were labeled "this is not what we are". Namely we wanted to create a place for people to be able to fill in that sort of information, and debate it.
We should be facing truth about what we do, and listening outwardly for feedback on how we are doing.
When it comes to benchmarks, I really believe we must stay out of the business of doing anything other then regression testing. I welcome everyone, and anyone else to do benchmarks.
I ask for groups to work with us so that we can improve upon what is found (and trust me... there is still plenty for us to find).
So the lack of comparison benchmarks has nothing to do with Sun, is has more to do with my own requests.
Drizzle as a project is about providing a new Kernel for MySQL, and I really believe others can do the benchmarks better then we can.
Those not attached to the core project come off as a legitimate source of information in a way that we simply can not, and should not, try to be.
On that same note... the Drizzle Project doesn't produce an OLTP engine. At some point we will get into the game of publicly comparing engines, but when we do it we will not be pulling any punches. We will be working with everyone we test with to make sure our final reports are as accurate as possible. I feel pretty free to compare libraries/plugins/engines that we make use of in Drizzle
It is only through public scrutiny that we can really provide distributions that bundle Drizzle with solid decisions on what to package.