Brian "Krow" Aker (krow) wrote,
Brian "Krow" Aker

Drizzle, The Death of FRM

One of the things about supporting multiple storage engines is that there is "no one way" to say "this is what we have".

Michael Widenius solution was to create a file called an "FRM" for MySQL that would work as a one to one ratio between table information and engine information.

All of the first engines were "stupid". Meaning that, the kernel handled lots of heavy lifting for them. As engines like Innodb were added, which were a lot more advanced, there was some friction.

Innodb keeps its own data dictionary, and so have a lot of the engines that have come after it.

This creates a very nasty problem, and a set of limitations. You can't do a lot of online work, because of the nature of ownership of "this is the real information".

And in the case of a crash? You can end up with orphaned tables or corruption on ALTER TABLE (which is very hard to make happen... but it can).

One of the ideas I pushed several years ago was flipping this around, letting the engines handle their own data.

Instead of trying to centralize data, we instead federate data from multiple engines. This allows engines to be their own bosses.

And as of yesterday? Thanks to Stewart, we now have this end to end.

FRM is now dead. Sure, we left helper functions around so that if you want the upper end to handle it, it can (though even this will soon be pushed all the way to the engine), but now an engine can fully handle its own information.

Network engines can now fully synchronize (just ask anyone who does MySQL Cluster, how important this can be).

Engines can now do anything online.

And as soon as remove the bits that force transaction commit on DDL operation?

Any engine that can do transactional DDL, will have that opened up to them.

We can now operate with much, much, smarter engines.

This allows us to fast forward the data dictionary and information schema work to where we can remove the bottlenecks on them. This will allow me to clean up the executioner a bit more so that we can narrow the paths on execution (aka... less bugs).

FRM served us well for a very long time, the design allowed us to build a kernel that could federate multiple engines, which was new at the time.

We have been talking about getting rid of FRM since around 2003. I remember a drive up to northern Finland with Kaj Arnö, where we spent an hour talking about this. I, David, and MontyW have talked about this for years.

Finally :)
  • Post a new comment


    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded