Brian "Krow" Aker's Idle Thoughts

Amazon RDS, MySQL, Hmm?

Fri, 06 Nov 2009 08:18:19 GMT

When a name like Amazon gets into the business, everyone acts like everything is new and shiny again :)

Why, I can remember like it was, well... almost a decade ago when I first saw a hosting vendor get into the business of support MySQL databases.

The first vendor I remember? Rackspace. Why? Because way back when, they were the main sponsor for the second OpenSource Database Conference.

Looking through the options that Amazon provides, they do look pretty sharp. Database backups can be a pain, and providing snapshot based ones is an excellent idea. There was some talk of snapshotting when I was at the MySQL User Group this week talking about Drizze. The general consensus was that this was the superior way to do backups.

The other backup methods, like using a physical backup tool, certainly work (Innodb has sold one for years, and Percona has one that they provide). If logical backups are your thing you can always use mysqldump with Innodb to do an online backup (If you look through past blogs you can see how I use it and the distributed revision control system Mercurial to backup my databases and provide point in time recovery/selective restores).

Backup services to S3 are not new at all. Zmanda started to provide a product for backing it up years ago. Googling on "S3 mysql backup", I found several links for HOWTO to do it.

Looking at the API, I believe native driver support is available. I would be curious to find out if they are supporting just the ASCII protocol, or if they have enabled the binary one as well (which is an awesome way of crashing your database). I am going to assume these databases are all in a self contained instance. If not? Should be simple enough to crash you and your neighbor. It would be interesting to find out how many changes they had to make to the database (or still have to make... all of this reminds me that they were heavily recruiting at the MySQL User's Conference last year). It was pretty obvious back then that they were going to work on this service. There is no mention of SSL support. The SSL stuff built into MySQL would be a pain to make work, but I suspect they could with some hacking.

I wonder if libdrizzle's mysql bits will work? I suspect we will have to try that out. If so, anyone need sharding in their driver?

BTW if you follow their "use mysqldump" model for pushing data into the service, remember that --single-transaction will allow you to do a hot backup. There is no need to lock up your current database. I am still appalled at how few people know that. Years ago we should have renamed "mysqldump" to "mysqlbackup" and defaulted the settings for Innodb. Outside of licensing, I suspect "how to backup" my database was the most often requested question.

Oh, that and "are you worried now that Oracle has acquired Innodb?"

Renaming our dump tool to backup would have made for pretty slick marketing :)

Amazon's sizing numbers look good. They have hit the sweet spot for most users.

Monitoring is nice, but also, not that hard. If you have everything already in AWS I can see where the "one stop shopping" would be nice (or is that one-click?).

There is more you can find in an article by Jeff. Rightscale has an article up as well.

I'm really curious to find out what they have disabled in 5.1, I have a hard time believing they can support all of its features. Coming up with an upgrade path should be interesting to watch. I've seen no sign of any patches coming back from them, so I would assume they have rolled their own. This means either they will be on 5.1 for a long time, or will be spending a lot of engineering time maintaining their own version. (Most of the bigger MySQL shops do, so it is not all that surprising.) The number of folks on the planet that can really maintain custom versions well is pretty small. I don't believe Amazon has any of them on staff. I am sure though that they can find people who can figure it out.

I have seen mixed results when companies keep their own custom versions of MySQL. It can be pretty easy to code yourself into a corner when you don't play an active part in the community. 5.1 should provide many years worth of use for them though.

Kudos to Amazon. Having them run a database service won't provide the "high end" sort of usage that keeps the folks who tune databases in business, but there certainly are a lot of users for whom this type of service will work just fine.

At the very least this service will certainly up the ante for others in the MySQL hosting business.

Providing services like RDS and application is the future for cloud companies. It will be interesting to see what they will come up with next.

I'm surprised Memcached hasn't been done as a service yet, but perhaps that is why I have those SASL patches sitting in my inbox.

We still have a little while before Gearman will be announced :)

Drizzle, InfiniDB, Column Oriented Storage

Tue, 03 Nov 2009 17:44:13 GMT

I have been asked a number of times "do you think there is a need for a column oriented database in the open source world?"

The answer has been yes!

Users and vendors have asked me this question a number of times. The problem has been most of the vendors were interested in creating closed source solutions around either Drizzle/MySQL, or, did their efforts in a way that made serious modifications to the backend (aka... made poor use of the storage engine interface).

For these reasons I have not really found myself all that thrilled to work with what has been out there. Also, I would often find that the commitment to open source was either luke warm or "we will do it, once we have some traction...".

My response to that? "Tell me more when you open source it. I'll see if it will work."

For this reason I was very happy to see Calpont do their release of Infinidb last week.

So as of this weekend?

We have a project to use their engine with Drizzle. Infinidb makes use of the storage engine interface I worked on for MySQL which is a subset of the interface we have built for Drizzle. We have had several engines ported already, but this will be the first column oriented engine we will have ported to Drizzle.

Building in different engines beyond the basic transactional engines is fun, because we get to see how the design stretches to fit additional needs. The core of Drizzle stays the same, but the micro-kernel nature of our design allows for others to expand the reach of where Drizzle can be used. Padraig started working on the engine on Friday and had it loading by the end of the weekend.

It should be fun to see what additional enhancements we can do out of the box with Infini engine :)

Note on sidewalk

Tue, 03 Nov 2009 16:37:13 GMT

Power Tools and Pumpkin Cutting

Sat, 31 Oct 2009 23:10:56 GMT

DSC_0189.JPG, originally uploaded by krow.

Game, Which is faster?

Thu, 29 Oct 2009 19:45:18 GMT

Last night while working with someone I am tutoring we got into a conversation of "what is faster". Below is some assembler that was created from it, for two different programs which did exactly the same thing, but in different ways.

The question is, which is faster?
Second Question, which do you think was the more maintainable code?

	.file	"?"
	.text
.globl main
	.type	main, @function
main:
.LFB0:
	.cfi_startproc
	pushq	%rbp
	.cfi_def_cfa_offset 16
	movq	%rsp, %rbp
	.cfi_offset 6, -16
	.cfi_def_cfa_register 6
	subq	$2097184, %rsp
	leaq	-2097184(%rbp), %rax
	movl	$1048576, %edx
	movl	$64, %esi
	movq	%rax, %rdi
	call	memset
	movl	$0, -20(%rbp)
	jmp	.L2
.L5:
	leaq	-1048608(%rbp), %rax
	movq	%rax, -16(%rbp)
	leaq	-2097184(%rbp), %rax
	movq	%rax, -8(%rbp)
	jmp	.L3
.L7:
	nop
.L3:
	movq	-8(%rbp), %rax
	movzbl	(%rax), %edx
	movq	-16(%rbp), %rax
	movb	%dl, (%rax)
	movq	-16(%rbp), %rax
	leaq	1(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$1, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	2(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$2, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	3(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$3, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	4(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$4, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	5(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$5, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	6(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$6, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	movq	-16(%rbp), %rax
	leaq	7(%rax), %rdx
	movq	-8(%rbp), %rax
	addq	$7, %rax
	movzbl	(%rax), %eax
	movb	%al, (%rdx)
	addq	$8, -16(%rbp)
	addq	$8, -8(%rbp)
	leaq	-1048608(%rbp), %rax
	addq	$1048576, %rax
	cmpq	-16(%rbp), %rax
	jne	.L7
.L4:
	addl	$1, -20(%rbp)
.L2:
	cmpl	$4095, -20(%rbp)
	jbe	.L5
	movl	$0, %eax
	leave
	ret
	.cfi_endproc
.LFE0:
	.size	main, .-main
	.ident	"GCC: (GNU) 4.4.1 20090725 (Red Hat 4.4.1-2)"
	.section	.note.GNU-stack,"",@progbits

	.file	"?"
	.text
.globl main
	.type	main, @function
main:
.LFB0:
	.cfi_startproc
	pushq	%rbp
	.cfi_def_cfa_offset 16
	movq	%rsp, %rbp
	.cfi_offset 6, -16
	.cfi_def_cfa_register 6
	subq	$2097184, %rsp
	leaq	-2097184(%rbp), %rax
	movl	$1048576, %edx
	movl	$64, %esi
	movq	%rax, %rdi
	call	memset
	movl	$0, -20(%rbp)
	jmp	.L2
.L5:
	leaq	-1048608(%rbp), %rax
	movq	%rax, -16(%rbp)
	leaq	-2097184(%rbp), %rax
	movq	%rax, -8(%rbp)
	jmp	.L3
.L4:
	movq	-8(%rbp), %rax
	movzbl	(%rax), %edx
	movq	-16(%rbp), %rax
	movb	%dl, (%rax)
	addq	$1, -16(%rbp)
	addq	$1, -8(%rbp)
.L3:
	leaq	-1048608(%rbp), %rax
	addq	$1048576, %rax
	cmpq	-16(%rbp), %rax
	jne	.L4
	addl	$1, -20(%rbp)
.L2:
	cmpl	$4095, -20(%rbp)
	jbe	.L5
	movl	$0, %eax
	leave
	ret
	.cfi_endproc
.LFE0:
	.size	main, .-main
	.ident	"GCC: (GNU) 4.4.1 20090725 (Red Hat 4.4.1-2)"
	.section	.note.GNU-stack,"",@progbits

"Fish" Foot Massage

Sun, 25 Oct 2009 00:37:20 GMT

When someone asks "Do you want to get a fish foot massage?", do not assume you just aren't hearing them well and that all you will be receiving a plain old foot massage.

Yes, those are fish in the picture, and yes they are eating the dead skin from my feet.

This is the first time I ever wanted to keep fish around the house. It tickles, but once you get past that?

Awesome :)

Guess!

Fri, 23 Oct 2009 04:39:55 GMT

RMS, GPL, The Peculiar Institution of Dual Licensing

Thu, 22 Oct 2009 18:28:52 GMT

I am going to take the words of my colleague Tim Bray to explain my own position regarding the acquisition of Sun by Oracle:


[Disclosure: I have no non-public information on any of the MySQL-related 
aspects of the Sun/Oracle transaction, nor on the current anti-trust review, 
and I am not speaking for anyone but myself.]

Just to further add to this, I am one of the main contributors to the largest community driven and contributed fork of the MySQL server, Drizzle.

From Richard Stallman:

MySQL is made available to the public in two parallel ways. Most users 
obtain it as free/libre software under the GNU General Public License 
(GPL) version 2; the code is released in this way gratis. 
MySQL is also available under a different, proprietary license for a fee.
This approach was able to provide (1) an attractive platform for 
developers looking to use FLOSS, and secured MySQL enormous 
mind share, particularly in supporting content rich web pages and 
other Internet applications, and (2) the ability for paying clientèle to 
combine and distribute MySQL in customizations that they do not 
want to make available to the public as free/libre software under the GPL. 
With excellent management and considerable trust within the user 
community, MySQL became the gold standard for web based FLOSS 
database applications.

What I find sad is Richard's commitment to the expansion of more close source software by the means of a "Robin Hood" sort of philosophy. Dual Licensing is nothing more then the creation of yet more software, where the end result is the creation of not more open source software, but the creation of yet more closed source software.

The GPL's approach is to provide a stick or carrot. If you are open source, then you don't pay, assuming of you are "the right" sort of open source. If you are close source or pick a license which is not compatible with the GPL you are forced into paying for use of a commercial license. When you "pay" for open source the freedom that was originally offered to the end user is removed. In the case of incompatible open source licenses you too are forced into the position of removing the freedom granted both and possibly the freedom you granted to your own work. Take any current distribution of a linux distribution and do the research on licenses within it. The tangled mess that is found will confuse anyone. MySQL itself was only able to solve some of this by offering a "FLOSS Exception" to the portions of the code it owns.

Dual licensing is anti-open source. I have no issues with this per-se, but I find his support of a "peculiar institution" a bit head scratching.

At the heart of dual licensing is "ownership rights", which inevitably come into conflict with the nature of open source. Open Source projects that preserve the ability to do dual license come into conflict with the developers who contribute the code. For the project to continue it must ask the original developer to give up their rights to the code via copyright assignment (there is some debate on whether copyright can be held in joint, but this is often disputed by lawyers). Thus dual licensing forces any developer who wishes to contribute into a position of either giving up their rights and allowing their work to end up in commercial software, or creating a fork of the software with their changes. In essence it creates monopolies which can only be broken via forking the software.Forking software over small changes is for the most part unviable because of the cost of keeping a fork of the software up to date, but it is not impossible.

I, being one of the main contributors of the largest of the MySQL forks, Drizzle, can tell you that this is no simple task but is quite achievable. Our reward has been that in our single year of operation we have achieved a larger base of contributions then MySQL achieved in its decade long existence. Community contribution at the expense of proprietary extensions is a small price compare if you consider the value that surrounds us by releasing that opportunity.

The conflict inherent in trying to reserve rights, and take the rights from others, leads to conflict.

If you wish to observe this in action, take a look at the internals@mysql.com list. Watch the jockeying and counter posture attacks that go on between Sun, MontyAB, and other groups. Each group trying to take rights and create position within the MySQL codebase. None of this does any good for the ecosystem surrounding MySQL. Users lose out in the end as bug fixes and enhancements are lost over an attempt to hold eventual rights that will be reserved for proprietary software, and at the same time creating a monopoly for the single party who gains "rights" control over the project.

What I find sad about this is Richard's own lack of imagination when it comes to the opportunity he has at hand. Here is an opportunity for him to influence Europe's main body and push forward the concepts of co-operation and community that surround open sources, and he wastes it for the opportunity to preserve the "peculiar institution" of dual licensing and furthering monopolistic intentions.

Preservations of the rights of end users, linking issues, open source license compatibilities, the rights of open source developers to continue their work... there are a plethora of concepts and actions he could be investigating.

Richard's advocation for dual license is in conflict with the concept of the nature of the free exchange of ideas and concepts within the open source community. His attempts at being an open source advocate on the point in question?

Fail.

libmemcached, protocol handlers, rolling your own server!

Thu, 15 Oct 2009 17:26:03 GMT

One of the cooler things has been worked on in libmemcached lately is the ability to tinkertoy up Memcached servers.

What does that mean?

There are a lot of forks of memcached. Some commercial, some open source, some that are in house at a variety of companies. They all share a few common concepts though.

Protocol

Scale Out

The recognition of this lead me to want to create a set of libraries that anyone, could use to build these types of servers quickly. After talking to a number of vendors and users it became pretty apparent that something like this needed to be done. The goal is to make it easy for people to write protocol compatible servers in a way that they can focus on the problems that they are concerned with, without getting bogged down in the details.

Also, for those of us who are client writers, we didn't want to see a tower of babble pop up around the protocol. memcapable, our protocol certification tool, works hand in hand with this to make sure that services that are being built today are compatible with the clients that have been released. No one benefits from a tower of babel.

Users want to know they can use different servers with existing applications, client writers want to be able to support users, and vendors of all variety want end user confidence in their solutions.

In 0.34 "libmemcachedprotocol" has now been added to libmemcached. Here is an example of the structure:

void initialize_interface_v0_handler(void)
{
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_GET]= get_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_SET]= set_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_ADD]= add_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_REPLACE]= replace_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_DELETE]= delete_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_INCREMENT]= arithmetic_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_DECREMENT]= arithmetic_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_QUIT]= quit_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_FLUSH]= flush_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_GETQ]= get_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_NOOP]= noop_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_VERSION]= version_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_GETK]= get_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_GETKQ]= get_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_APPEND]= concat_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_PREPEND]= concat_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_STAT]= stat_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_SETQ]= set_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_ADDQ]= add_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_REPLACEQ]= replace_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_DELETEQ]= delete_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_INCREMENTQ]= arithmetic_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_DECREMENTQ]= arithmetic_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_QUITQ]= quit_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_FLUSHQ]= flush_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_APPENDQ]= concat_command_handler;
   interface_v0_impl.interface.v0.comcode[PROTOCOL_BINARY_CMD_PREPENDQ]= concat_command_handler;
}

As you can see, the interface is a designed around callbacks. You supply the callbacks, we handle the protocol parsing, both binary and text. The idea is to let you hook up the memcached protocol to anything. If you download the latest code, or just grab via CMS, aka "bzr clone lp:libmemcached", you can get a feel for it.

Take a look inside of example/interface_v0.c and you will find a simple single threaded memcached server that uses just a simple linked list to store items held in memcached.

Want something more complex?

Take a look inside of example/storage_innodb.c and you can see where we have an example memcached storage system using the Innodb Embedded Engine. Coming soon, a longer article on this!

Wanting to contribute one for Tokyo Cabinet, BDB, SQLite? Please send patches :)

Right now we are still accepting feedback on the interface. While we have shopped it around and gotten some great feedback, I would love to see more. We have not finished writing the encapsulation functions just yet, so we haven't generated API for Ruby, Perl, Python, etc just yet, but I expect to see this happen over the next couple of months.

Have fun with it!

New in Drizzle, 2009.10.1178

Tue, 13 Oct 2009 16:22:33 GMT

Welcome to the metronome, every two weeks we drop a release of Drizzle.

Which makes it sort of boring to blog about each release since they are consistent :)

But no two releases are really the same, and this last one had a bit of fun in it:

drizzle> CREATE TABLE A (a int) ENGINE=Archive;
Query OK, 0 rows affected (0 sec)

drizzle> create table B like A ENGINE=innodb;
Query OK, 0 rows affected (0 sec)

drizzle> show create table B;
+-------+-----------------------------------------------------------+
| Table | Create Table                                              |
+-------+-----------------------------------------------------------+
| B     | CREATE TABLE `b` (
   `a` int DEFAULT NULL
) ENGINE=InnoDB |
+-------+-----------------------------------------------------------+
1 row in set (0 sec)

Being able to specify an engine when doing a CREATE TABLE LIKE has been long on the list of things to do to make the server more orthogonal for a while now. You would think it would have been easy to do, but it never really was. In MySQL we didn't have a data dictionary so a command like the above was just a "copy the frm file in place/tell the engine it exists". Which does work, but it didn't make it very simple to manipulate. Sure, it could have been done, but it wasn't all that simple.

For Drizzle we just ask the StorageEngine factory "find the engine with this definition and had it back to me as a proto". Once we have the proto we just make any changes we want to it, and then tell the StorageEngine factory "go create this". After that? It gets sent to whatever engine is supposed to create the table and the table gets created. How the data is stored is up to the engine. The advantage of the federated data dictionary is that engines own their own table information. No 2PC is required for creation and network based engines are free to track their own changes. Pretty handy :)

Another thing that is changing is our parser. Drizzle's parser no longer knows or confirms engines. It simply parses the statement and lets the execution system validate the state or existence of the engine (this is making way longterm for the future where parsers just parse... validation belongs elsewhere in the execution pipe). Our execution system no longer exists as a giant switch/case system. Instead we just generate an object and pass that along for execution. A big advantage we were able to realize because of this was that all of our DDL structures are no longer included in the global Lex structure.

Gearman functions were updated with this release, so we have up two date functionality happening with our integration there. Replication is continuing to evolve and as of this version we now have information schema tables that report on the state of whatever memcached cluster we are attached too. Memcached support that is built into the server didn't make it in this release, but I expect to see it in next week's.

Latest version of Bell, our current release milestone, can be found here:
https://launchpad.net/drizzle/+download

Stargate Universe is Space 1999

Wed, 07 Oct 2009 06:37:49 GMT

Yes I have been hearing from folks "its Voyager" or "its BattleStar Gallactica".

Wrong! It is Space 1999.

Now for all you who were raised with X-Wing fighters, you should acquaint yourself with the world of the Eagle Transporter. It will be left as an exercise for the reader to go acquaint themselves via wikipedia with the background of the show.

I could claim that the nuclear explosion which sent the moon hurtling away from the earth in Space 1999 forms the opening point to why the shows are twins born decades apart, but that would be basing everything on a very weak argument (...no more then say, the moon being knocked out of orbit and thrown at lightspeed away from the Earth).

In Stargate Universe you have a Stargate Crew which is hurtling through space on a starship they cannot control, much like the crew of the moon in Space 1999. Who can possibly control a moon that has been blown out of its orbit? Each week the crew is going to come into contact with a stargate that they will be able to access for a very limited period of time. They will be able to get onto planets once they come into dialing distance, but they will have to get back to ship before they are out of range.

In Space 1999 you had crews running off on Eagle Transporters once the moon came into distance of a planet (yes... ignore completely the physics of this). They would have a limited time to examine the planet before the moon would go out of range of their craft to join back up with it.

I am wondering if we can just line the episodes up, and play "match that episode".

The main scientist for Stargate Universe outfit would easily fit into the 1970s.

I wonder how many episodes it will take before they reach some sort of Dystopian world?

Alien telepaths?

With both BSG and Atlantis going off the air I thought for sure I would be getting to take a year off from poorly produced and thought out Science Fiction.

Oh well! :)

Go, Go, Conferences... Drizzle, Memcached, and Gearman

Wed, 07 Oct 2009 01:37:20 GMT

So where will I be at over the next couple of months?

In October I will be hitting the Malaysian Open Source Conference:
http://foss.my/

In November I will be making my way to Portland for the OpenSQL Camp:
http://opensqlcamp.org/Main_Page

In January I will be attending Linux.Conf.Au
http://www.lca2010.org.nz/

In Malaysia I have scheduled talks on Drizzle, Gearman, and Memcached. For SQL camp there will be more of the same :)

Looking for more information in Gearman? Check out this article by Rasmus Lerdorf:
http://toys.lerdorf.com/archives/51-Playing-with-Gearman.html

For Memcached we have released the memcapable program with the latest libmemcached. This will allow us to test vendors for compliance with the new binary protocol (and as of the next release the ASCII as well). This should lower some headaches on all sides. As developer we will have an easier time diagnosing problems with servers that aren't built on the reference implementation, and for end users they can be assured that the products they are buying are following the protocol standard. For vendors? They will be able to create products knowing that the end user can drop their software stacks/appliances in place and be up and running quickly.
http://blogs.sun.com/trond/entry/memcapable

More updates to follow... we have been busy at work on all of the projects in the last couple of months :)

Bowling!

Tue, 29 Sep 2009 05:01:14 GMT

Decibel Festival

Mon, 28 Sep 2009 00:06:28 GMT

Source Lake

Sat, 26 Sep 2009 22:00:53 GMT

Tunnel Hiking Failure

Sat, 26 Sep 2009 19:18:08 GMT

Drizzle, Optimizer, Execution Flow

Mon, 21 Sep 2009 16:29:34 GMT

The present in my RSS feed this morning was an article by Peter Zaitsev on :
http://www.mysqlperformanceblog.com/2009/09/20/guidance-for-mysql-optimizer-developers/

One of the changes made recently to Drizzle, is a redesign of our executioner. This was done by Padraig. MySQL inherited a design where the parser uses one Global Lex Structure to fill in the members of the query for later use. This structure has members in it for every form of query that can be executed. The structure is also assigned an ENUM that will be used later for determining if the query is a SELECT, INSERT, etc... via a switch/case dispatcher.

This of course creates a number of limitations in the design, especially for Drizzle since we focus on micro-kernel over monolithic design. What Padraig completed recently was work that took apart the switch/case dispatcher and replaced it with an object executioner. Each of the query types, SELECT, INSERT, etc,... are now assigned to a Statement object which are assigned in the parser on the top level. We are now focusing on removing the Global Lex Structure and placing its members into the Statement execution objects.

This gives us a smaller footprint parsing and it makes the entire system pluggable in that we can now truly break down the parser in ways that we couldn't before. Before to extend the parser you would have to grow the global lex structure for any extension to the SQL syntax/execution functionality. The global lex structure has a very large footprint and you have to weigh the cost of any new feature vs growing the size of the global structure. Setting the default values for this alone is fairly expensive (memset() is not your friend).

It also meant that for someone who wanted to write a new execution path, that had to modify the switch/case executioner and fiddle with all of the code along the path.

Now in Drizzle? Anyone who wants to write a new execution path can focus on just a single class. This class is generated in the parser and encapsulates all of the logic for a given statement. BTW these Statement objects are not limited to the SQL parser, so any of the new methods for Connection can generate one as well (without going to deep here, what I mean is the work that Eric Day has been doing that will allow us to speak native HTTP/etc will now be able to use direct REST parsers... so SQL is just one method of execution among many).

This move to Statement gives us a design which is less complex then what we did in the past and it allows a developer who wants to extend the database to focus on just what they want to work on. Hell, maybe we should put together a talk for the East Coast No-SQL conference just to make a point about flexibility around SQL vs No-SQL designs.

So why did I open up this article up with a reference to Peter's article?

Just as the switch/case executioner existed for all queries, a similar one existed for the optimizer. About two months ago we started straightening it out so that we could do the same to it as we have been doing to the main switch/case dispatcher. From the top down we are able to use similar designs at each connecting point in the database.

The gain is? Just as our Statement Executioner cracks the door open to the main executioning at the core of Drizzle being pluggable, by reuse of design the same will be true of our join optimizer (!!!). Once we are complete, our Join Exectioner will be extendable by just adding in new plugins. Our plans are moving along, and the pluggable parser is on our horizon. The goal will be a set of classes that can be extended so that the Join Executioner can be easily extended as well.

When I read articles like Peter's, and Peter is one of the experts in tuning MySQL, it makes me pretty excited to see validation in the concepts we hold dearly. I'm not sure that we will pull off exactly what Peter wants longterm, time will only tell, but I do see that the concepts that we are pushing are shared by others as well.

Sometimes it takes three...

Fri, 18 Sep 2009 23:47:00 GMT

DSCN0955, originally uploaded by krow.

The Mac Mini has now been "popped". Both the CDROM and the hard drive have now been replaced :)

Steve

Wed, 16 Sep 2009 20:11:23 GMT

IMG_8139, originally uploaded by bradyf.

One of my favorite installations from this year's Burning Man. Steve tracked people entering/exiting from it. It would assign people to instruments and then mix sound based on movement.

Pretty awesome.

Darwin's Temple

Tue, 15 Sep 2009 22:14:30 GMT

DSC_0038.JPG, originally uploaded by krow.

Notice that the "crash pads" have been moved to the side.

Off to see the Wizard... Wicked tonight!

Sat, 12 Sep 2009 03:01:24 GMT

Burning Man, Road Trips, Community

Thu, 10 Sep 2009 06:30:41 GMT

One flat tire and nearly a thousand miles round trip I am back in Seattle again!

It is really difficult to explain to others exactly what Burning Man is, in large part because it is so many things. Art exhibition, music festival, community, city, party...

Every day the event changes. Black Rock City grows from nothing, evolves into a city of tends of thousands, and then returns to nothing within a week.

No two days of the event are the same.

The early part of the week is nothing like the later part of the week.

The weekend?

Saturday's Man burn is completely different from the Temple burn that comes Sunday. The dichotomy between the two is incredible. It is hard to believe that the two occur at the same event, with many of the same people.

At the core is a concept of "radical self reliance". The concept is impossible to escape. It does not matter if you come in a tent or an RV, at some point you will face yourself, your fears, and your own expectations. No matter what happens, no matter if you overcome or fail, you come out differently on the other end.

Standing out on the playa one night, I spoke to a friend who had only just arrived midweek. She was wound up and the look in her eyes were one of apprehension.

I mentioned to her that no matter what, her being there meant that she was going to change.

Her response?

"That is what I am afraid of."

By the end of the week? She had a smile on her face and was talking about her experience in glowing terms.

That sort of transformation has been the common currency of the event for the people I have traveled with.

Community?

Everywhere you walk you are met with people greeting other people. Gift giving, sharing, and warm welcomes are everyday events. I have had people join our camp midweek. I have seen people show up at camp only to return at the end week. They were never seen between those two times.

Sleep is the enemy.

The hardest part of Burning Man?

Returning Home :)

Welcome to your next Hot Shower

Thu, 10 Sep 2009 05:00:51 GMT

Welcome to your next Hot Shower, originally uploaded by krow.

Yes, we had hot showers. Often scalding!

Worm Tube

Thu, 10 Sep 2009 03:22:19 GMT

DSC_0275, originally uploaded by krow.

This is a view down the "tube" of the structure I was working on for the last few months. Other then a few bad pieces of rope we replaced on the first day it turned out incredibly well :)

Now I just need to start working out what I will build for next year!

Cube

Tue, 01 Sep 2009 06:31:06 GMT