?

Log in

No account? Create an account

Mercurial, Clever or just a weird way to do Backups?

« previous entry | next entry »
Apr. 22nd, 2008 | 10:55 am

All of my sites run a piece of software called Everything. Mine is a fork from the original Everydevel Corp's release. It shares some common traits with Slash but it has a more rapid development cycle that I like.

One of the things it shares with Slash, is that the "site" is the database. AKA everything that makes a site, a site, is the database. This means that a backup of the site can be tossed into an everything container and be up and running in seconds (I use this with virtual machines so that I can move stuff around on a whim).

I hate backups. I don't want to take snapshots with LVM, since I know I just need the content, and typically with dumps I end up with a huge set of these files just sitting around.

  • I cannot search them.
  • I cannot diff them between dates.
  • I cannot partially restore them.
  • They take up a lot of space

    So it dawns on me at dinner last night. What if I just used Mercurial?

    So I add a crontab like so:

    0 3 * * * mysqldump -f --single-transaction -T /var/backup/sitename sitename; (cd /var/backup/sitename; hg commit -m "auto"; hg push)


    I push the backups to a centralized server.

    Instant daily backups, and I can partially restore thanks to the tab format. I can pull diffs that I can directly apply and reinsert into the database.

    Perfect? Nope.

    Mercurial is storing deltas so the space difference does not seem to be to bad (and I am only shipping what changed over the wire). This will not work for my Archive tables, they are just too damn big (4 billion rows and growing). Those tables are just logs and I have a different long term plan for them.

    But the data that makes up the site? Works well.

    Since I use Innodb for sites, I can use the --single-transaction trick to take an online backup. If you are using any other storage engine, the above crontab will lock up your site while the dump is being made.

    This will probably work for most revision systems (if you take one of the nifty ones that can chop history you could even automate the tossing of deltas after a week or so). Mercurial has the strong benefit of being http centric and it is incredibly easy to install as a server (and it has never eaten my data like some other open source systems have).

    I welcome feedback :)
  • Link | Leave a comment |

    Comments {16}

    (no subject)

    from: sykosoft
    date: Apr. 23rd, 2008 05:35 am (UTC)
    Link

    One of the things that really made me happy was that with an existing (non lvm, etc) system, with mixed table types (innodb, myisam), we could just install the r1soft agent, and make a backup of mysql, in real time, with no table locks or downtime, it's incremental, and I can selectively restore certain tables. I've had the (unfortunate) opportunity to test the restores, and they're very painless, even to an alternate server/location.

    Michael

    Reply | Parent | Thread

    Brian "Krow" Aker

    (no subject)

    from: krow
    date: Apr. 23rd, 2008 06:13 am (UTC)
    Link

    What all did you have to install to make it work?

    Reply | Parent | Thread