February 26th, 2007

glasses

Table Discovery, Memcached... and how does this get solved?

Somewhere in the last week I got a simple request, a user wanted table discovery for the memcached engine.

Both the Archive and Cluster engines support this.

Take an archive .ARZ table in 5.1, move it to another server. If you SELECT or INSERT off the table name it will be "discovered" immediately. The FRM just appears and everything just works (which means that yes in 5.1 all you have to do is save the ARZ file and you are set... no need for the FRM). Using archive_reader you can even make an online copy of the table (this was a feature request from one of the Livejournal developers last year at the user's conference).

Cluster has a similar mechanism. If you create a table on one front end MySQL server, all other MySQL servers that are talking to that cluster discover the table. The reason that this is possible is that the MySQL server just speaks to one cluster at a time.

The memcache engine can talk to many clusters, hell, it can talk to as many as you want on a per table basis. It does not have this limitation.

This is what makes table discovery difficult, there isn't just one cluster of memcached servers to talk too.

Its possible to setup one server and say "this is the dictionary server", and then have all servers look to that server. It would also be possible to iterate over each found server and poke at each memcached cluster looking for the table definition.

...or just make everyone type create table themselves on each server...

The older I get the lazier I get, I just want stuff to work.
glasses

Seattle Viaduct, Big Dig, Prefer ZigZag

As I mark "No" on my ballot for both options for replacing the Seattle Viaduct, I ask myself the following question.

If we are even thinking about digging a big hole in the ground for the Viaduct, why not just knock it down and instead bury I-5 instead?

The Real Estate that would be gained would be nicer. The mess that I-5 creates in the city would be gone, and the ambient noise in the city would go down by some big factor.

Why bother with the Viaduct, lets bury I-5 instead!

I also want a Monorail for Christmas...
...and a cupcake.
glasses

Memcache Engine for MySQL version 0.5

The two big changes in 0.5 are that I fixed a memory leak and it now has its own information schema table on statistics.

For example:
mysql> select * from information_schema.memcache_servers;
+----------------+-------+---------------+-------------+-------+---------------------+-------------------+-----------------------+------+------+------+--------+------------+---------------+----------------+
| NAME | COUNT | CURRENT_ITEMS | TOTAL_ITEMS | BYTES | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS | CONNECTION_STRUCTURES | GETS | SETS | HITS | MISSES | BYTES_READ | BYTES_WRITTEN | LIMIT_MAXBYTES |
+----------------+-------+---------------+-------------+-------+---------------------+-------------------+-----------------------+------+------+------+--------+------------+---------------+----------------+
| localhost:6688 | 2 | 5 | 39 | 399 | 2 | 65 | 3 | 236 | 39 | 197 | 39 | 4669 | 9546 | 67108864 |
| localhost:6666 | 2 | 4 | 12 | 311 | 1 | 62 | 3 | 143 | 12 | 114 | 29 | 2215 | 5977 | 67108864 |
+----------------+-------+---------------+-------------+-------+---------------------+-------------------+-----------------------+------+------+------+--------+------------+---------------+----------------+
2 rows in set (0.01 sec)

The tables shows the current status of any memcached server that is attached to MySQL. There are a few more statistics that I can grab but the assortment I picked for the first version should be very useful. The schema is such:

mysql> desc information_schema.memcache_servers;
+-----------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+--------------+------+-----+---------+-------+
| NAME | varchar(120) | NO | | | |
| COUNT | bigint(4) | NO | | 0 | |
| CURRENT_ITEMS | bigint(4) | NO | | 0 | |
| TOTAL_ITEMS | bigint(4) | NO | | 0 | |
| BYTES | bigint(4) | NO | | 0 | |
| CURRENT_CONNECTIONS | bigint(4) | NO | | 0 | |
| TOTAL_CONNECTIONS | bigint(4) | NO | | 0 | |
| CONNECTION_STRUCTURES | bigint(4) | NO | | 0 | |
| GETS | bigint(4) | NO | | 0 | |
| SETS | bigint(4) | NO | | 0 | |
| HITS | bigint(4) | NO | | 0 | |
| MISSES | bigint(4) | NO | | 0 | |
| BYTES_READ | bigint(4) | NO | | 0 | |
| BYTES_WRITTEN | bigint(4) | NO | | 0 | |
| LIMIT_MAXBYTES | bigint(4) | NO | | 0 | |
+-----------------------+--------------+------+-----+---------+-------+
15 rows in set (0.00 sec)

There is still quite a bit of work to be done but it seems to be working without issue. Next on the list is to support multiple indexes per tables.

For other engine writers you can check out how to add information schema via this patch:
http://hg.tangent.org/memcache_engine?fd=9bc58f383d06;file=src/ha_memcache.cc

You can download 0.5 from here:
http://download.tangent.org/memcache_engine-0.5.tar.gz

When you upgrade you will have to run this command to get the information schema:
mysql> INSTALL PLUGIN memcache_servers SONAME 'libmemcache_engine.so';
glasses

ExploitSeattle, Rocky Horror, Everything

I've been working on a new backend for ExploitSeattle! during my last couple of vacations. It will be based on a new copy of everything that I have been working on. The old site is based on Slash which is a problem since:

1) Slash won't work on anything other then Apache 1.3
2) I dislike how Slash uses memory.
3) I don't feel like hacking on it anymore (and have not in about four years).

Number three is the most important reason :)

I've been using my custom copy of edev for TangentOrg, Krow.net, and a few other sites I maintain for a few years. The only site left on Slash is ExploitSeattle and I am hoping to finally move it in about a week or so.

So why the reference to the Rocky Horror in the subject? I started using Google Analytics back in October to get statistics on sites. Its rather slow but it works (though it does under report...).

The page most referred from Google according to Analytics? The page with the schedule for the Rocky Horror Picture show and its schedule in Seattle. The second most visited page? A link to some Christian Rock show that passed through Seattle a while ago.

Funny :)