Drizzle, State of Testing

« previous entry | next entry »
Jun. 4th, 2009 | 10:05 am

Testing, Testing, Testing...

I've gotten a number of questions about how we are doing testing, and even how our methodology for accepting code works :)

A lot of this comes from running open source projects for almost a couple of decades (hell, if I toss in uploading public domain to software to BBS'es for the Commodore 64 it is a bit longer!).

One of the most important rules I have learned over the years is that anything that is not automated and not required, will get skipped.

Today Drizzle runs 213 tests, the entire MySQL test suite minus tests that are for features we don't have. We don't allow for any regression, meaning that no one is allowed to disable a test in order to get their code pushed. Our test suite was also modified so that we can run all of the tests against a particular engine. Today we do this with both Innodb and PBXT. So instead of having "engine specific" tests, we can test everything. Feedback we are getting from storage engine vendors is that this is golden. Even if they never release for Drizzle, they can use it to vastly increase the testing they do today.

We also do not allow any code to be pushed that causes a compiler to toss a warning. We do this for a wide set of versions of gcc, and also for Sun Studio. We treat warnings as errors :)

We are also enormously proud of this fact. This took a lot of effort :)

Our most recent change is that we now include a regression test for performance for sysbench. For each push into our "staging" tree we run a full test at different steps of "connections". We test both read-only and read-write workloads. My only real complaint right now about this system is that we look at absolute numbers, and being a math geek, I would really like some more information on standard deviation :)

The development process we use maximizes our use of tools. For Drizzle we accept no patches via email. Your patch must come through Launchpad, where you must have an account. We do this so that we can always track down "who did what". This is done almost entirely without exception, and the exception being that I am sure someone at some time as either pointed out a bug in code to me or has made a comment on how something should be written while reading over my shoulder.

We don't require anyone to sign a "all your bases belong to us" sort of contract. Speaking as myself, I personally find them unpalatable and frequently suggests developers think about them before ever signing. Code flowing is what I find important. With Drizzle we have had a 100+ contributors, and I suspect that number would be much smaller if we had taken a different point of view on this. One of the most interesting conversations I ever had about this topic was with Rasmus, who has continued to refuse to sign the Apache Foundation's agreement. If someone was to ever put code in from their employer, we can strip the code back out in seconds. We can also hand over all of their information to the original copyright holder for them to prosecute. The ones of us who do final review of code are very nitpicky about this. One of the nice things about "source code" is that it has finger prints. Anything that doesn't follow our coding style, which is very specific, stands out.

Code which was not original tends to always show it roots. I know from talking to Theodore Tso that IBM gives its developers who take in code to its open source projects, a class in how to identify copyright violations. IBM's general handling of open source always tends to impress me.

Having a modular approach in our design also means that any "large" sort of change can be reduced down to a plugin. Our average patches are bug fixes or refactoring bits. If you find that you enjoy making code readable, fast, and standard looking, you will probably enjoy working on Drizzle.

If you want to write fun and new code, then you should look at writing plugins!

On a personal note, if I was to write a database from scratch, I would go after a completely different problem domain. I still happen to need a relational database though, which is why I work on Drizzle.

That and the people who are working on the project are both awesome and fun :)

Code flow in Drizzle is pretty simple. You write a patch, you submit a tree to Laundpad, and one of the captains reviews it and puts it into their tree. I pull from one of their trees and merge into a local tree. Before I push the code it has to pass all tests/etc on 64bit Intel Fedora, Open Solaris Sparc, Solaris Sparc, and OSX. When I get around to buying myself a new desktop Mac I am going to spin up a few more platforms in virtual boxes, so that I can test more platforms. There is a simple Gearman system I use to populate and run the code on all of these systems.

Once the code passes on all of the above it goes to our staging tree on launchpad. From there the automated system gives me feedback on regression. If we pass then the code goes to trunk. Our turn around time for code is frequently about 24 hours. Since all of the above testing is done, we drop tarballs anytime we want too.

It is pretty much clockwork for us. If a human wasn't involved I suspect we could just set a cronjob to handle it every two weeks (and who knows... if Lee gets bored with this, maybe he will do exactly that!).

What is the future?

Today we generate automated reports for cachegrind, callgrind, and valgrind. We run pahole by hand. We are told that there is a new tool for generating random queries for MySQL for crash testing, we need to look into this.

I've also got a set of tests written around drizzleslap (aka mysqlslap) that we need to toss in soon.

All of these need to go into the process. We should never regress on L2 misses or branch predictions. We should never see holes in our structures/classes.

We don't have enough tests for failure cases. Our test coverage is public:
http://drizzle.org/lcov/index.html

I want to see that increased. The problem is that there has never been a test system for "this is how to fail at X". A few hacks have been written, but we need to come up with a complete methodology for this. The one open source database that has an awesome test suite for this is SQLite, and we need to spend some time learning from them all of what they do.

Over dinner recently with Josh Berkus, he mentioned the work they are doing on pgbench. I am hoping that we can get a patch in it so that it will support libdrizzle. That way we could use one of the Postgres tools as well.

These are our next big steps :)

Have any suggestions? Want to contribute? All of our tools are open source. We welcome extensions and they are general enough that almost any other database could use them as well!

Link | Leave a comment | Share

Comments {15}

Documentation for our Automation Suite

from: jaypipes.myopenid.com
date: Jun. 4th, 2009 05:44 pm (UTC)
Link

Just a quick link for those interested in the automated regression system. It is documented on our wiki here, and I will continue to add more documentation:

http://drizzle.org/Automation_Documentation

Cheers, and nice write up!

Jay

Reply | Thread

kartar

Re: Documentation for our Automation Suite

from: kartar
date: Jun. 5th, 2009 03:08 am (UTC)
Link

Link is 404'ed.

Reply | Parent | Thread

Re: Documentation for our Automation Suite

from: jaypipes.myopenid.com
date: Jun. 5th, 2009 04:53 pm (UTC)
Link

Doh!

http://drizzle.org/wiki/Automation_Documentation

Thanks!

Jay

Reply | Parent | Thread

(no subject)

from: anonymous
date: Jun. 4th, 2009 06:30 pm (UTC)
Link

For Drizzle we accept no patches via email. Your patch must come through Launchpad, where you must have an account. We do this so that we can always track down "who did what". This is done almost entirely without exception.


The last sentence contradicts the first two sentences.

Reply | Thread

Brian "Krow" Aker

(no subject)

from: krow
date: Jun. 4th, 2009 06:41 pm (UTC)
Link

Thanks for pointing that out. I have clarified my statement.

I don't believe I live in a bubble. People interact with me all of the time and that influence is of course found in the code. I've also seen patches in mailing lists that I have looked at and rewritten, though that has only happened once.

People seem to be pretty open to using bzr.

Reply | Parent | Thread

System for users and partners to automatically load test cases

from: anonymous
date: Jun. 4th, 2009 07:18 pm (UTC)
Link

Brian,

Now I think you and the Drizzle posse need to build a system where users can add their own test cases.

LS

Reply | Thread

Brian "Krow" Aker

Re: System for users and partners to automatically load test cases

from: krow
date: Jun. 4th, 2009 07:22 pm (UTC)
Link

We setup a submission system for new tests... but so far zilch :(

Reply | Parent | Thread

SQLite

from: dmarti
date: Jun. 4th, 2009 09:24 pm (UTC)
Link

How SQLite Is Tested has some good testing info too. "The SQL Logic Test or SLT test harness is used to run huge numbers of SQL statements against both SQLite and several other SQL database engines and verify that they all get the same answers. SLT currently compares SQLite against PostgreSQL, MySQL, and Microsoft SQL Server. SLT runs 7.2 million queries comprising 1.12GB of test data."

Reply | Thread

King InuYasha

Re: SQLite

from: king_inuyasha
date: Jun. 5th, 2009 03:50 am (UTC)
Link

Holy crap, that is a ridiculously large amount of queries and data...

It also doesn't sound too realistic, but meh...

Reply | Parent | Thread

Request your project to be added to Coverity's Scan site

from: anonymous
date: Jun. 5th, 2009 12:36 pm (UTC)
Link

If you want an automated tool which can show "this is how to fail at X" type reports - check out http://scan.coverity.com - and add yourself as an open source project.

Reply | Thread

Brian "Krow" Aker

Re: Request your project to be added to Coverity's Scan site

from: krow
date: Jun. 5th, 2009 02:43 pm (UTC)
Link

Interesting tool... though I can't figure out from the page how to sign up :)

Reply | Parent | Thread

Re: Request your project to be added to Coverity's Scan site

from: dmarti
date: Jun. 5th, 2009 03:29 pm (UTC)
Link

their FAQ says to mail scan-admin@coverity.com .

Reply | Parent | Thread

Brian "Krow" Aker

Re: Request your project to be added to Coverity's Scan site

from: krow
date: Jun. 5th, 2009 03:50 pm (UTC)
Link

Found it, and did :)

Though... a web page without a web form? Hopefully they are working on improving the web interface. The actual product is pretty good.

Reply | Parent | Thread

Andy Lester

(no subject)

from: petdance
date: Jun. 7th, 2009 07:47 pm (UTC)
Link

Glad to see that it's hurtling ahead.

So has any of the testing infrastructure that I was working on been of any use? I can only hope.

Reply | Thread

Brian "Krow" Aker

(no subject)

from: krow
date: Jun. 7th, 2009 08:36 pm (UTC)
Link

No one has finished up anything, so it languishes. S far no one has been able to build a new system to completion.

Reply | Parent | Thread