Brian "Krow" Aker (krow) wrote,
Brian "Krow" Aker

Giuseppe's Article on Partitioning

Just finished reading this:

His comments on partitioning affecting Archive are dead on. If you can keep it from scanning then it works better (though if all engines have to scan it normally is a big factor of improvement over the other engines).
Archive's compression cannot do much with a single column table like he is using, for that it has to do too much effort (and while there is optimizations for this, I would rather just write a column oriented store some day to solve these sorts of problems).

It does make me wondering how those tests would come out with 6.0. He has a multi-processor box, and in 6.0 Archive uses multiple threads for IO to achieve a bit better performance. I don't have any good benchmarks right now, so I am not really sure.

What is in the back of my head when I think of Partitioning and Archive?

I wonder if I should not just pull it natively into the engine since I could do more with it there. The partition engine is just a simple wrapper around the Engine interface. Most of the hard logic is in the optimizer work that Sergey Petrunia did to allow pruning, so it is not a ton of work to support it natively.

Pushing down the pruning information would allow me to do more with parallel IO (and would allow me to make more operations online).

No time at the moment to work on this, but it is something to think about.
  • Post a new comment


    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded