Drizzle, The Death of FRM

« previous entry | next entry »
Jul. 29th, 2009 | 10:05 am

One of the things about supporting multiple storage engines is that there is "no one way" to say "this is what we have".

Michael Widenius solution was to create a file called an "FRM" for MySQL that would work as a one to one ratio between table information and engine information.

All of the first engines were "stupid". Meaning that, the kernel handled lots of heavy lifting for them. As engines like Innodb were added, which were a lot more advanced, there was some friction.

Innodb keeps its own data dictionary, and so have a lot of the engines that have come after it.

This creates a very nasty problem, and a set of limitations. You can't do a lot of online work, because of the nature of ownership of "this is the real information".

And in the case of a crash? You can end up with orphaned tables or corruption on ALTER TABLE (which is very hard to make happen... but it can).

One of the ideas I pushed several years ago was flipping this around, letting the engines handle their own data.

Instead of trying to centralize data, we instead federate data from multiple engines. This allows engines to be their own bosses.

And as of yesterday? Thanks to Stewart, we now have this end to end.

FRM is now dead. Sure, we left helper functions around so that if you want the upper end to handle it, it can (though even this will soon be pushed all the way to the engine), but now an engine can fully handle its own information.

Network engines can now fully synchronize (just ask anyone who does MySQL Cluster, how important this can be).

Engines can now do anything online.

And as soon as remove the bits that force transaction commit on DDL operation?

Any engine that can do transactional DDL, will have that opened up to them.

We can now operate with much, much, smarter engines.

This allows us to fast forward the data dictionary and information schema work to where we can remove the bottlenecks on them. This will allow me to clean up the executioner a bit more so that we can narrow the paths on execution (aka... less bugs).

FRM served us well for a very long time, the design allowed us to build a kernel that could federate multiple engines, which was new at the time.

We have been talking about getting rid of FRM since around 2003. I remember a drive up to northern Finland with Kaj Arnö, where we spent an hour talking about this. I, David, and MontyW have talked about this for years.

Finally :)

Link | Leave a comment | Share

Comments {5}

Justin Swanhart

OMG. Transactional DDL. AWESOME.

from: swanhart
date: Jul. 29th, 2009 06:12 pm (UTC)
Link

Sorry to sound like a teenager again, but really, I've been wanting transactional DDL for a very long time.

The Google protobuf library which makes this FRM removal possible is really neat. I've introduced our engineers to protocol buffers because we currently maintain XML and other structures which would be much better represented with protobufs instead.

I'm learning tons already from just a short amount of time following the Drizzle community.

Thanks guys.

--Justin

Reply | Thread

Brian "Krow" Aker

Re: OMG. Transactional DDL. AWESOME.

from: krow
date: Jul. 29th, 2009 06:31 pm (UTC)
Link

The Google Proto code has just been awesome for us.... its not that it does anything special, its just that it works and has multi-language support.

Reply | Parent | Thread

Nice work

from: lphuberdeau.com
date: Jul. 29th, 2009 06:38 pm (UTC)
Link

All this refactoring work around Drizzle is truly inspiring for any of us having to work every day with legacy code. Every story is like a breath of fresh air.

Can't wait to start using it.

Reply | Thread

Kytty

(no subject)

from: kytty
date: Jul. 29th, 2009 08:07 pm (UTC)
Link

Nice Work! Keep it up :)

Reply | Thread

(no subject)

from: joelpietersen
date: Feb. 25th, 2010 03:19 pm (UTC)
Link

Interesting stuff indeed.

Reply | Parent | Thread