Brian "Krow" Aker's Idle Thoughts

Drizzle, Rethinking MySQL for the Web (Video from OSBridge)

Tue, 30 Jun 2009 16:21:31 GMT

Here is the link to the video of the talk I gave at OSBridge ->

http://blip.tv/file/2296093

It is the standard "this is Drizzle Talk".

Rosalynd

Sun, 28 Jun 2009 01:05:57 GMT

DSC_0022, originally uploaded by krow.

Gratuitous pictures of my dog :)

Shade Structure - Mark I Design

Sun, 28 Jun 2009 01:03:23 GMT

P1000547, originally uploaded by krow.

A bunch of photos from our adventure today in setting up part of the new shade structure this year for Burning Man.

Today? Successful?

Yes, but we learned a lot.

Next week?

We move on to Mark II design. I'll write more about all of this later. I just wanted to get some photos up :)

And so it begins....

Sat, 27 Jun 2009 20:23:54 GMT

Lullaby Moon

Tue, 23 Jun 2009 07:19:23 GMT

DSC_0387, originally uploaded by krow.

From this month's http://www.lucianeare.org/lullabymoon.htm

Regression, The Bummer

Wed, 10 Jun 2009 19:14:26 GMT

Wahoo!

We finally found the regression problem in Drizzle that we have been looking for over the last couple of months.

In the processes of doing this was have walked every line of code. I sat the other night doing a single step through the entire sysbench run looking for anything out of place. Nothing came up at all.

Eric was the person who finally asked the question "could it be tcmalloc"? No one had assumed it because it typically is a good solution (and we will be looking into why it turned out to be at fault, we will probably push now to more aggressively remove the MEMROOT system we inherited since we suspect it/it doesn't play well with C++).

We have not been able to push any patches in the last couple of months that really fixed other performance issues that we know exist.

Why?

Because we feared complicating the problem of finding the original problem. We have all spent time looking through our ancestors to see if there was something we missed.

1) Could it be C++?
2) Could reducing the number of locks, creating a traffic jam around a single lock?
3) Was UTF-8 at fault?

In the end it was none of these :)

So for us?

We have patches coming soon to optimize the UTF-8 system, to minimize LOCK_open, optimize/simplify the THR lock system simpler, and to partition caches internally.

Stored Procedures or Server Side Scripting?

Wed, 10 Jun 2009 18:24:34 GMT

Here is a bit of code I worked up for us a recently for Drizzle:

drizzle> DELIMITER |
Note that there is no semicolon after the '|' symbol, which we will use as the delimiter for our purposes. You have to choose a delimiter that does not appear in your procedure, and it can be more than one character.

drizzle> CREATE PROCEDURE perl_hello (param1 string)
-> return "Hello " . $_[0] . "!"
-> |
Query OK, 0 rows affected (0.05 sec)

drizzle> CALL perl_hello('Brian');
-> |
Query OK, 1 row affected (0.00 sec)

drizzle> DELIMITER ;
drizzle> SELECT @perl\G
*************************** 1. row ***************************
@perl: Hello Brian!
1 row in set (0.00 sec)

Stored Procedures!?!

In an actual language!?!

About a week ago I was talking to a CTO for a company who is looking at adoption of Drizzle. One of things he came back with was "I don't need stored procedures, but I do need server side scripting".

Back at the very first MySQL User's Conference we had a debate over the future of stored procedures in MySQL. I and some others really wanted the first stored procedure language to be external, David really wanted it to be PHP. I didn't see the value in implementing a single language. I thought people would be more interested in writing code in whatever language they wanted. Also, I figured that an external system would allow for different groups to develop languages more rapidly.

Fast forward to when we began Drizzle. Parsers are where you spend a lot of your time. The smaller the parser the better off you are. So I went to task removing all of the signs of the SP language from Drizzle. We have been free of them now for over a year now (yes, long before we went public). Things are finally shaping up so that when we begin on Bell, our next milestone, stored procedures, or something like them, are now on our list.

Though are they stored procedures, or is this server-side scripting?

A few premises of the design:

Any language should be pluggable. We won't have a native language.

We only support re-entrant engines. This solves all of the pre-locking problems that exist currently. We haven't removed 40% of the locks in Drizzle, only to have to come up with a bunch of new ones to support engines which were never built to handle this stuff.

While it won't be required, we will focus first on enabling scripting languages that are not in-process. Why is that? We don't want anyone to crash the database. Informix had this problem early on and got a bad rep for it. We want to avoid this.

We will enable driver writers to be able to communicate in a native way. AKA if you are writing something in Java, you will be able to use a JDBC interface inside of the database. For Perl DBI, etc. I want to be able to test my SP's in any environment. The difference between running in of the database, and out of should be trivial or non-existent.

I am a little bit torn about using the SP call/creation SQL commands in Drizzle. You won't be doing the typical SP language (well... unless someone wants to write a plugin for them!). I would also like to encourage people to think differently about what writing server side code should look like. Personally I don't feel that stored procedures are the right solution for a lot of the cases, keep your business logic in your application layer(!), but we also know that users expect to be able to be able to run code locally. Triggering/Callback mechanisms can be very useful though, and enabling them is a part of this. Doing Triggers today in C is simple, but that is not something that everyone should/would/could want to do.

Putting this in the plugin structure means no overhead to the parser or the rest of the database. Keeping them out of process means no drain or memory expansion of the Database. SMP boxes will benefit because you can confine the language VM to a particular set of processors/amount of memory.

We don't want the database to ever blow up because of bugs in the execution language!

And if you never want them? You never load the plugin in the first place.

Why Perl? I've embedded Perl for years and know how to make it work. I've only done Java once, so I will leave that to other experts.

I suspect I can find a Java person somewhere inside of Sun :)

Drizzle, State of Testing

Thu, 04 Jun 2009 17:05:56 GMT

Testing, Testing, Testing...

I've gotten a number of questions about how we are doing testing, and even how our methodology for accepting code works :)

A lot of this comes from running open source projects for almost a couple of decades (hell, if I toss in uploading public domain to software to BBS'es for the Commodore 64 it is a bit longer!).

One of the most important rules I have learned over the years is that anything that is not automated and not required, will get skipped.

Today Drizzle runs 213 tests, the entire MySQL test suite minus tests that are for features we don't have. We don't allow for any regression, meaning that no one is allowed to disable a test in order to get their code pushed. Our test suite was also modified so that we can run all of the tests against a particular engine. Today we do this with both Innodb and PBXT. So instead of having "engine specific" tests, we can test everything. Feedback we are getting from storage engine vendors is that this is golden. Even if they never release for Drizzle, they can use it to vastly increase the testing they do today.

We also do not allow any code to be pushed that causes a compiler to toss a warning. We do this for a wide set of versions of gcc, and also for Sun Studio. We treat warnings as errors :)

We are also enormously proud of this fact. This took a lot of effort :)

Our most recent change is that we now include a regression test for performance for sysbench. For each push into our "staging" tree we run a full test at different steps of "connections". We test both read-only and read-write workloads. My only real complaint right now about this system is that we look at absolute numbers, and being a math geek, I would really like some more information on standard deviation :)

The development process we use maximizes our use of tools. For Drizzle we accept no patches via email. Your patch must come through Launchpad, where you must have an account. We do this so that we can always track down "who did what". This is done almost entirely without exception, and the exception being that I am sure someone at some time as either pointed out a bug in code to me or has made a comment on how something should be written while reading over my shoulder.

We don't require anyone to sign a "all your bases belong to us" sort of contract. Speaking as myself, I personally find them unpalatable and frequently suggests developers think about them before ever signing. Code flowing is what I find important. With Drizzle we have had a 100+ contributors, and I suspect that number would be much smaller if we had taken a different point of view on this. One of the most interesting conversations I ever had about this topic was with Rasmus, who has continued to refuse to sign the Apache Foundation's agreement. If someone was to ever put code in from their employer, we can strip the code back out in seconds. We can also hand over all of their information to the original copyright holder for them to prosecute. The ones of us who do final review of code are very nitpicky about this. One of the nice things about "source code" is that it has finger prints. Anything that doesn't follow our coding style, which is very specific, stands out.

Code which was not original tends to always show it roots. I know from talking to Theodore Tso that IBM gives its developers who take in code to its open source projects, a class in how to identify copyright violations. IBM's general handling of open source always tends to impress me.

Having a modular approach in our design also means that any "large" sort of change can be reduced down to a plugin. Our average patches are bug fixes or refactoring bits. If you find that you enjoy making code readable, fast, and standard looking, you will probably enjoy working on Drizzle.

If you want to write fun and new code, then you should look at writing plugins!

On a personal note, if I was to write a database from scratch, I would go after a completely different problem domain. I still happen to need a relational database though, which is why I work on Drizzle.

That and the people who are working on the project are both awesome and fun :)

Code flow in Drizzle is pretty simple. You write a patch, you submit a tree to Laundpad, and one of the captains reviews it and puts it into their tree. I pull from one of their trees and merge into a local tree. Before I push the code it has to pass all tests/etc on 64bit Intel Fedora, Open Solaris Sparc, Solaris Sparc, and OSX. When I get around to buying myself a new desktop Mac I am going to spin up a few more platforms in virtual boxes, so that I can test more platforms. There is a simple Gearman system I use to populate and run the code on all of these systems.

Once the code passes on all of the above it goes to our staging tree on launchpad. From there the automated system gives me feedback on regression. If we pass then the code goes to trunk. Our turn around time for code is frequently about 24 hours. Since all of the above testing is done, we drop tarballs anytime we want too.

It is pretty much clockwork for us. If a human wasn't involved I suspect we could just set a cronjob to handle it every two weeks (and who knows... if Lee gets bored with this, maybe he will do exactly that!).

What is the future?

Today we generate automated reports for cachegrind, callgrind, and valgrind. We run pahole by hand. We are told that there is a new tool for generating random queries for MySQL for crash testing, we need to look into this.

I've also got a set of tests written around drizzleslap (aka mysqlslap) that we need to toss in soon.

All of these need to go into the process. We should never regress on L2 misses or branch predictions. We should never see holes in our structures/classes.

We don't have enough tests for failure cases. Our test coverage is public:
http://drizzle.org/lcov/index.html

I want to see that increased. The problem is that there has never been a test system for "this is how to fail at X". A few hacks have been written, but we need to come up with a complete methodology for this. The one open source database that has an awesome test suite for this is SQLite, and we need to spend some time learning from them all of what they do.

Over dinner recently with Josh Berkus, he mentioned the work they are doing on pgbench. I am hoping that we can get a patch in it so that it will support libdrizzle. That way we could use one of the Postgres tools as well.

These are our next big steps :)

Have any suggestions? Want to contribute? All of our tools are open source. We welcome extensions and they are general enough that almost any other database could use them as well!

Folk Life

Tue, 26 May 2009 07:34:19 GMT

Folk Life, originally uploaded by krow.

Photos fro Folklife.

Ridge Poles

Tue, 26 May 2009 07:03:16 GMT

DSC_0137, originally uploaded by krow.

A few pictures of the ridge pole design for our main structure for Burning Man this year. A piece of rebar that the PVC is being seated over is what keeps the pole suspended in the air :)

Sleepers on the Beach

Mon, 25 May 2009 04:17:13 GMT

Libmemcached, BZR, Launchpad

Wed, 20 May 2009 18:43:37 GMT

Today I moved from using Mercurial to using bzr on Launchpad for libmemcached.

Why BZR?

I use Launchpad for pretty much all of my projects at this point. I have been really happy with it, and I have found that bzr works well for Linux/Windows/Mac. Drizzle, Gearman and others are already there so this just simplifies my daily workflow.

This should help with me being able to take patches a bit more quickly (and for that matter do reviews). Having contributors push their patches to their own trees, allows me to easily pull patches and do reviews. With Drizzle we keep a "staging" tree just so that we can regression test any code before it goes to trunk. Since code in Drizzle goes through several people before I see it, we each get to make revisions on it and look for issues like copyright violations/etc. By tracking code by Launchpad account, we always know who code came from.

For libmemcached I will also be enabling the bug system on Launchpad. While tracking via email has worked, I would like to come up with something a bit more formal.

I've been using Laundpad for over a year now and I am really happy with it. Career wise I have used several proprietary systems, of which Bitkeeper was the best, and several open source ones, of which Subversion was the worst (it was the only system that ever lost code).

All of the distributed revision control systems are pretty good now. I've used all of the major ones and have not really had many issues. The reason I go with bzr really has to do with workflow with Launchpad.

Mercurial is great, and I will continue to use it for a lot of personal projects. It is the only open source distributed revision control system that I find to be easy to host. Its ability to do push/pull from HTTPS is a major asset for me. So for work I do where I need to continue to host the data I will be sticking with it. I completely understand why Google went with it, and I really wish BZR would add this feature.

Looking back over the last few years it is amazing to see how far the revision source control system have gone. Just a few years ago all of the tools were garbage and it was a nightmare to pick one.

Thankfully we are well past that.

Libmemcached Version 0.29 Released

Tue, 19 May 2009 15:54:49 GMT

We have released version 0.29 of libmemcached.

The highlights:
* Fixed malloc usage to calloc for spots where we need zero filled memory.
* All code warnings now treated as errors.
* Fixes for debian packaging.
* Added new pooling mechanism.
* MEMCACHED_BEHAVIOR_NO_BLOCK no longer also sets MEMCACHED_BEHAVIOR_BUFFER_REQUESTS.
* Updated generic rpm.

The new pooling mechanism is thread safe so if you have been looking for a way to limit your number of memcachd_st structures and you use threads you are now set!
This work was done to improve the connection handling in Drizzle for Memcachd (and it is going into the UDF for MySQL as well).

You can find out more information here:
http://tangent.org/552/libmemcached.html

Mushroom hunting success!

Sun, 17 May 2009 03:49:50 GMT

Memcached, Speaking of Performance...

Wed, 13 May 2009 22:40:56 GMT

I've been looking at the cost recently of fetches for libmemcached, and seeing if there was anything I could do to increase performance.

What did I find? Half the cost currently is in the snprintf() sitting at the protocol sender.

Half!

Good news about this?

If you are using the new binary protocol with Memcached this goes away completely :)

snprintf() sucks.

Innodb Concurrency, No Kidding...

Wed, 13 May 2009 21:22:04 GMT

Back in 2007 I, and then several others, pointed out the Innodb Concurrency should be set to zero (or any large number) if you are planning on scaling.

http://krow.livejournal.com/542306.html

I don't get why anyone is congratulating themselves on this self discovery, since we have all known about this for over two years. It is great to see that the default setting for 5.4 is finally being fixed but this shouldn't be a revelation to anyone who has been working with the server for years (and the Innodb team fixed this in their own plugin a while ago, which should be no surprise since they author the code). This was one of the first, and frankly simple, changes made to Drizzle.

Here are some more obvious ones:

Kill the query cache (and disable Stored procedures and their cache if you can).

Remove the locks around show process list.

Fix the binlog (or just ditch it).

Fix all of the locking in the access controls.

Increase the default table cache.

Depending on your filesystem/etc move to multi file Innodb (for that matter, take the Innodb plugin for a spin for compression).

...

Want the best bang for the buck for performance?

Buy SSD for your databases, and go figure out how to use a caching layer so as to not need to manage a bunch of replication servers.

If you are worried about managing your caching server I would look at Gear6 or Virident. I am especially impressed with the Gear6 replication technology.

Books, Libraries, Victory?

Mon, 11 May 2009 16:17:47 GMT

Victory!

Well sort of.

This morning I finally completed the loading of all of my books into LibraryThing.

The reason I started this project was to keep myself from buying duplicates (and to keep down the number of duplicates within my friend circle in general).

The problem is, most of the books I don't care about really. Other then graphic novels, the rest are for the most part paperbacks I just loan out. Some I get back, some I don't. I've given away scores of O'Reilly books and dozens of copies of a couple of other series. When I find some books for under a buck I just buy them so that I can give them out to others.

It is not like I own every book I have ever read.

What I would love to have is a dump of the database from my childhood libraries (or from the library in Iowa City, I read a lot as a graduate student). Part of me would love to know some of the titles of books I have almost completely forgotten. This sort of information would be fun to have.

As for books I currently have?

The graphic novels I worry about duplicating since I pick up a lot of these, almost all, at used bookstores. So it is handy to pull up a browser and sort through a stack of them before I go to the checkout counter.

Everything else?

If I was to move nowadays I would shed a lot of books before I did. Less and less, do I see the point of keeping them around. There are only a few technical books that have stayed relevant enough for me to want to keep them. For fiction, once I have read the book I am done with it. I hardly ever go back and reread fiction.

If it wasn't for DRM, I would just switch to electronic books (and I fully expect DRM to go away with books within the next few years).

The value of an object is inversely valued based on a ratio of its weight and awkwardness when carried up a flight of stairs. When I stopped moving every year I stopped tracking this as much, but I still ponder this fact whenever I buy something (I live on the third floor of an old house with a very narrow stairwell). Books weigh a lot when you have to carry crates of them.

The main realization of this morning when I completed cataloging all of the books?

I think I still have a crate of Geology and Math textbooks downstairs that I have never unpacked.

So much for completion.

Drizzle, Regression, STL Bitset

Sat, 09 May 2009 00:15:09 GMT

So how is the regression issue coming along?

Glad you asked!

Earlier in the week Jay found that the bitset patch gave us significant regression. This particular patch was a move from using a built in bitmap system MySQL had, to using the bitset found in the standard template library.

Since then we have been able to find issues with it, but no conclusion on how to solve it via a new design/container/etc.

So what are we doing?

Earlier this week I refactored the interface to the Table objects to create, well, an interface!

It is not perfect but it encapsulated a large chunk of the code. Monty today put back into the original MY_BITMAP but did so behind this interface. Jay has run the numbers and declared that the regression can no longer be found.

What will happen from here?

We are going to work on the interface some more. Basically make it so that we can change the back end to the bitmap without changing a lot of code.

Right now we have a couple of ideas on how to solve the problem (I favor a bool in Field object, Monty wants to look at vector, Mats has a bitvector). We will test each of these and find a solution that gives us in the end better code with no regression.

Solving this issue we had to look at a number of things. Our methods, the outcome of a tree rollback, and if performance in this case mattered. The problem wasn't simple, and all solutions had draw backs. The main thing we were not going to do was push some code which caused regression that we would then "find a solution for in the future". That was not acceptable. Rolling back the tree? We could do this, I favored it if we had no other solution, but we determined that we could patch up the tree without causing this sort of disruption.

And?

Encapsulating the interface gives us room to find a new solution.

So what came out of all of this?

We are moving to a staging tree.

As of now we have an lp:drizzle/staging tree. This tree should not be pulled from. We will send code here before it is sent to trunk. If code fails the performance regression testing then we will pull it back.

So what does this mean? No more pushes from main that bypass staging. We have pointed the automatic regression testing at this tree. I am going to be suggesting that we point Hudson and Buildbot at it as well. If a tree can pass here then it will be moved to the main tree.

And what are the thoughts on regression for the future?

Jay asked me today "what do we mean by regression?". To me we can calculate regression pretty easily. We look at the standard deviation of all previous runs and apply it to the current tree. If we find that we are within norms then the new code is fine (and I suspect we will refine this formula in the future). This was my suggestion.

But what if regression happens and there is an argument for letting it happen?

Then we talk about it on the mailing list. Right now most of us have seen the numbers showing that 5.4 is faster then Drizzle at 16 concurrent connections. We have been looking into this, but we may find that some of the decisions that let us scale out to more connections/processors contributed to this. That is ok. Our target is not the 16 connections sites, it is the sites that need mass numbers of connections/threads/processors. If we find a change that hurts us at 1-N and N is a small number that may be ok.

What will we do when we are confronted by this? We talk abut it on IRC and we will send the information to the mailing list.

More eyeballs is a good thing.

Innodb Embedded Engine

Wed, 06 May 2009 18:15:47 GMT

A couple of notes:

1) I have already shared these thoughts with the Innodb team (and received some encouraging thoughts).
2) I wrote this about a week ago. Drizzle/Memcached/Gearman keep me busy so I only got to spend a couple of weekend days to look over the Innobase Embedded Engine. I wish I had more time to go in depth.
3) Innodb has a forum for the engine here where you can get more answers: http://forums.innodb.com/list.php?8
4) You can download the technology from here: http://www.innodb.com/products/embedded-innodb/
5) I'd love to see someone take the embedded engine and port it directly to the Drizzle Engine Interface. I think that the interface they have done would make a much better starting point for integration then what we have today.

Technology

The Innodb Embedded Engine is the same technology used for the Innodb Plugin. That is awesome... think about it. You are getting a threaded storage engine for embedded use that understands schema. Schema-less certainly has its place but there are a lot of cases where schema matters. Better? We are talking about using a real OLTP engine and being able to skip all of the SQL and go straight to the interface. A number of the storage engines vendors have talked about doing this over the years, but this is the first time one of them as delivered.

All of the settings that you can do from the normal MySQL/Drizzle configuration are available. You are exposed to the direct API so you can get at the bits if all you really need is a simple interface (joins are not supported).

Observations

How are the examples?

Without a bit of work they won't compile. There are problems in the include declarations for paths.

Simple stuff like this:
/usr/local/include/embedded_innodb-1.0/api0api.h:26:23: error: ib0config.h: No such file or directory

The paths were incorrectly setup in the header file. My suggestion to the authors would be to look at libxml2 to see how to properly setup header files.

Also? Don't include the config file. You break everyone else who is using auto tools. If you see errors like this you know you will have issues:

/usr/local/include/embedded_innodb-1.0/ib0config.h:348:1: warning: "PACKAGE_NAME" redefined

The naming conventions are pretty poor. For instance the include file names make no sense:
api0api.h db0err.h ib0config.h

The naming scheme is reflective of Innodb's history but for end-developer usage they could have picked something a little bit cleaner (unless you are a Solid DB author, since they use exactly the same naming conventions) .

I noticed when writing code that I must insert as a table name "something/something". You are required to build names like this:
snprintf(table_name, sizeof(table_name), "%s/%s", dbname, name);

From what I gather from the assert the "/" is required for the embedded plugin to know the name of a schema to create.

The error system lacks some sort of error to string function. Having this would make debugging much simpler.

I may be wrong about this though. The examples are a bit confusing and the call ib_database_create() makes me think that there is more to explain then what the current manual has in it. The documentation is far above the quality of what is frequently found for a first release like this... kudos to Oracle for doing this. The only issue I found was a discrepancy in the ib_cursor_moveto() function and a few other random mistakes.

Unlike SQLite, the Embedded Innodb requires multiple files to run. Depending on your use case this will be annoying. The "single file" feature of SQLite really makes it useful as a replacement for writing your own file formats.

The library makes use of its own types, like ib_bool_t, instead of just using standard C99 types. This type of programming is currently a pet peeve of mine. I am getting tired of dealing with "yet another set of types". It makes it a pain to integrate with other code and just increases complexity for the end developer for no good reason.

And writing to stdout on startup? That is a no no. I don't want my libraries writing to stdout on me. Libraries should be quiet, not noisy.

Errors like this make the point that it is not only noisy, but that it still isn't really ready for prime time yet:

090425 14:38:31 InnoDB: Error: table test/Foo does not exist in the InnoDB internal
InnoDB: data dictionary though the client is trying to drop it.
InnoDB: Have you copied the .frm file of the table ?

or...

090425 15:36:52 InnoDB: Error: Client is freeing a thd

That remind you of anything? :)

The API is based on what has been needed so far to make Innodb work with MySQL, so MySQL above specific errors are not surprising. The lack of API calls to list all tables in a schema is another example of this. With MySQL you had the FRM files, so there was no need to be able to get at the list of tables Innodb owned, so no API call exists for this (which is a common need in a standalone database).

One other big item, this library is using a lot of global variables. Take a look at ib_cfg_set_int(). Notice the lack of context for setting variables. This means that the library really was not meant to be used in any sort of multi-tenancy use case. This really limits its usage pattern. A better design would be to create a "context" and pass that to the startup of the library. I've wanted Innodb to cleanup its usage of global variables for years, and I had hoped that with the creation of a library this would have been solved.

Final Thoughts

I can see a lot of use for this library. Concurrency with SQLite is a big issue for write, and libraries like Berkeley DB lack schema, and I personally like the concept of an embedded engine knowing more about the data then the length of the byte array being stored in it.

Still? The library could use a lot of polish. It still shows the warts of being an internal project that has been pushed out into the public. I really hope that before a final release occurs that the authors will clean up the interface and consider the end developer.

As far Drizzle/MySQL goes? I can certainly see basing a future storage engine interface around the concepts found in this library. It is not perfect but it is certainly better then what we have today. It is also obvious after looking at the interface that there is more that can be done in regard to performance if the interfaces were better aligned.

There is a lot of potential in this project.

Benchmarks, Comparing Drizzle to others

Thu, 16 Apr 2009 00:21:17 GMT

"I was surprised for example to hear from Jay that the Drizzle team should not compare its performance to MySQL only to Drizzle itself"

This quote came from a piece of email today asking me about why we, as in the Drizzle project, have not been publishing comparison benchmarks.

First of all, this is not a Sun request, this was a request from me to the other core members of the project.

Let me explain...

I don't think projects can ever objectively compare themselves to other projects. Look at all of the heat that went on for years between Postgres and MySQL. I think we, the Drizzle Project, are better off publishing code and helping others create benchmarks, but avoiding doing the comparisons ourselves.

We are inherently biased.

One of the first things that went up on the Drizzle wiki were the pages that were labeled "this is not what we are". Namely we wanted to create a place for people to be able to fill in that sort of information, and debate it.

We should be facing truth about what we do, and listening outwardly for feedback on how we are doing.

When it comes to benchmarks, I really believe we must stay out of the business of doing anything other then regression testing. I welcome everyone, and anyone else to do benchmarks.

I ask for groups to work with us so that we can improve upon what is found (and trust me... there is still plenty for us to find).

So the lack of comparison benchmarks has nothing to do with Sun, is has more to do with my own requests.

Drizzle as a project is about providing a new Kernel for MySQL, and I really believe others can do the benchmarks better then we can.

Those not attached to the core project come off as a legitimate source of information in a way that we simply can not, and should not, try to be.

On that same note... the Drizzle Project doesn't produce an OLTP engine. At some point we will get into the game of publicly comparing engines, but when we do it we will not be pulling any punches. We will be working with everyone we test with to make sure our final reports are as accurate as possible. I feel pretty free to compare libraries/plugins/engines that we make use of in Drizzle

It is only through public scrutiny that we can really provide distributions that bundle Drizzle with solid decisions on what to package.

Up at Index

Fri, 10 Apr 2009 01:50:03 GMT

Niagra, Daily Work

Mon, 06 Apr 2009 18:57:03 GMT

First off, I have not been much of a of for Sparc in a number of years. Partly because I didn't feel like paying for the machines, and partly because I am not much of a fan of Solaris (the user space for development has a lot to be desired). In the last few months we have been working on getting Drizzle to work on Sparc/Solaris which has been an interesting adventure.

MySQL never compiled on Solaris without warnings, and it almost never could pass the full regression test system.

With Drizzle, I felt like if we were going to say we ran on Sparc/Solaris that I wanted to be able to say that we supported it without a bunch of exceptions.

It took us months... and I mean months to get things cleaned up for it. In the process I've gained a bit more respect for the platform (keep in mine... I was a big fan of SunOS, the predecessor to Solaris).

A few nice things I have noticed:

Compiling on the Niagra machine is fast. I run gmake -j and the thing just plows through the compile. Its faster then my 8proc Intel and way faster then my Mac Mini. What is impressive about this is that the Sparc is doing the compile with the write cache not enabled, while Linux and OSX are (hell... OSX has a fake sync() call, how lame is that?)

The Sun Studio compiler consistently finds more bugs/gives me better warnings then gcc. Sure, it lacks the built in atomic calls that gcc has, but thanks to working with the Sun Studio team we have been able to make use of their atomic.h and make a very simple compatibility layer. We don't handle our own assembler, we let the experts do that.

The more we reduce locks, the faster it gets. I am still arguing with some of the authors of the other MySQL type trees about this... there is still a concept that some locks are ok. I am finding that every turn even the most innocuous are still an issue. As an example, there is a lock around the query string for "show proccesslist". That one particular lock has turned out to be a real bane in production environments where you have lots of clients logging in and out... especially when you toss in monitoring. It took a few weeks to get rid of the locks there but we have them all gone now. It made a noticeable difference for some of the live usages of Drizzle (aka... just beyond benchmarks).

All in all I am becoming a bit happier with the Sun/Solaris environment. I would prefer to just have Debian on the box, with an "apt-get solaris-kernel" to test against. It would give me a better apples to apples comparison. Valgrind and a number of other tools would be nice... but in general?

I am finding that the environment is not as bad as I thought it would be to maintain, and that there are some actual benefits.

WebOps Speakers Club @ Web2.0 Expo

Thu, 02 Apr 2009 18:54:14 GMT

WebOps Speakers Club @ Web2.0 Expo, originally uploaded by jesse robbins.

Just how many patches does Drizzle take in?

Wed, 01 Apr 2009 17:35:21 GMT

This morning I woke up to an IM of "Just how many patches do you all take?"

I suspect the question was asked because of Masood's post about contributions for MySQL.

Looking at yesterday's commits I see that we took in contributions from four people. There were 26 patches in total from those four people, and of those we had to correct two of the patches. One of those people comes from a team of three at one company who are working fulltime on Drizzle at the moment in order to get their code released.

Only one actual paid for by Sun employee committed anything yesterday :)

Which reminds me that I need to get back to hacking, the global locks around @variables are not going to fix themselves :)

More fun numbers:

Last month?

47 contributors according to Launchpad.

In total sense the project became public last year?

Somewhere around a hundred people.

We have five active Drizzle developers in Sun right now (Lee has been too busy as of late with other stuff, hopefully we can change that!).

None of this includes work on the new drivers.

Web 2.0 Expo, Drizzle

Tue, 31 Mar 2009 16:47:55 GMT

Just as a reminder I will be giving a talk on Drizzle at Web 2.0 Expo in San Francisco this week on Thursday at 1:30.

http://www.web2expo.com/webexsf2009/public/schedule/detail/5782

I will be hanging around the conference for the day so feel free to say hello :)