?

Log in

No account? Create an account

Performance, It is about Caching Today

« previous entry | next entry »
Mar. 1st, 2008 | 08:35 am

CPU evolution has been about getting faster.

Toss in new instructions that do the work of four old instructions, into one.

Gamble on the outcome, branch prediction.

More layers of caching.

When I cook I take the ingredients out of the fridge and place them on the counter where I will do the prep. From there they move to the flat area next to the stove.

I am caching so I can make food more quickly.

In computing we do a lot of caching today.

This came out of the specs from a user who I have been talking to:

"80 nodes, 640 cores... with 21 nodes in our Lustre cluster serving 65T... which we can sustain ~2.5GB/s to"

Problem was?

It was not fast enough for their needs. The system worked well enough for "containing the data", but for executing on the data it could not keep up.

Solution?

Today they have a 100 nodes with 2gigs of ram a piece. They are using libmemcached to access their memcached servers on the nodes.

Classical web application?

No their business has nothing to do with the web :)

They do geo exploration.

Back a year ago there was a great quote, that I cannot find, from the CIO at Walmart.

What impression did it leave me with? That Walmart realized that they need to be more like a YouTube, and less like Big Iron 90's company.

Link | Leave a comment | Share

Comments {7}

Artur Bergman

(no subject)

from: crucially
date: Mar. 1st, 2008 05:36 pm (UTC)
Link

You have good experiences with lustre?

Reply | Thread

Brian "Krow" Aker

(no subject)

from: krow
date: Mar. 1st, 2008 05:40 pm (UTC)
Link

I have no personal experience. I can ask someone I know who has deployed it if that would mind me connecting you with them.

Reply | Parent | Thread

Artur Bergman

(no subject)

from: crucially
date: Mar. 1st, 2008 05:42 pm (UTC)
Link

Please do.

Thanks!

Reply | Parent | Thread

Brian "Krow" Aker

(no subject)

from: krow
date: Mar. 1st, 2008 06:22 pm (UTC)
Link

Mail sent, when he replies to me I'll drop you a note (he is down under so I suspect it will only take a day)

Reply | Parent | Thread

Tanjent

That reminds me

from: tanjent
date: Mar. 1st, 2008 05:41 pm (UTC)
Link

I never got back to you with that hash function I was talking about -

unsigned int murmurhash ( unsigned int * buffer, int count )
{
	const unsigned int m = 0xCEC16335;
	const int r = 4;

	unsigned int h = 1;

	for(int i = 0; i < count; i++)
	{
		h = h * m + buffer[i];
		h = _rotl(h,r);            // rotate left
	}

	for(int i = 0; i < 5; i++)
	{
		h = h * m;
		h = _rotl(h,r);
	}

	return h;
}


Works great, passes all the tests mentioned here. The 0xCEC16335:4 constants aren't magic, I just mutated a bunch of pairs until I found some that give good mixing behavior.

-tanjent (I'm going to Saturday House this week, I swear...)

Reply | Thread

Tanjent

Re: That reminds me

from: tanjent
date: Mar. 2nd, 2008 05:36 am (UTC)
Link

Don't use this, I posted an even better one that beats Hsieh and Jenkins hands-down.

Reply | Parent | Thread

Brian "Krow" Aker

Re: That reminds me

from: krow
date: Mar. 2nd, 2008 02:22 pm (UTC)
Link

Want to send me a patch for libmemcached :)

Reply | Parent | Thread