?

Log in

No account? Create an account

Building a Storage Engine: Getting the Skeleton to compile

« previous entry | next entry »
May. 1st, 2007 | 03:19 pm

This is the first part, in what hopes to be, a series where I go through the entire process of building a storage engine for MySQL.

The first thing to do is to download the skeleton engine. You can find the latest copy at:

http://hg.tangent.org/skeleton-mysql-engine?ca=tip;type=gz

You will need to untar and uncompress the file. The Skeleton engine is a basic bare bones engines which is a starting point for creating a storage engine project (the memcache, PBXT, AWS, HTTP, Google File system and many other engines all evolved from it).

What you will find once you download the engine and cd into its directory, is that it doesn't have a configure file just yet. To generate one, run the following:


[brian@zim skeleton-mysql-engine]$ sh config/bootstrap
+ aclocal
+ autoheader
+ automake --add-missing --copy
+ autoconf


The concept is that anything which can be generated, should not be checked into the source control system.

At this point you should now have a configure file. What you will now need to do is supply a source tree to compile against. It is important to look at src/Makefile,a before you do this. By default it is set to compile against a normal MySQL tree, but if you are building with a debug tree you will need to change the INCLUDE line in the Makefile.am.


# By default we are set to not compile with debug
INCLUDES = -DMYSQL_DYNAMIC_PLUGIN -DUNIV_MUST_NOT_INLINE -DFORCE_INIT_OF_VARS $(MYSQL_INC)
#INCLUDES = -DMYSQL_DYNAMIC_PLUGIN -DDBUG_ON -DSAFE_MUTEX -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DFORCE_INIT_OF_VARS -DSAFEMALLOC -DPEDANTIC_SAFEMALLOC -DSAFE_MUTEX $(MYSQL_INC)


This step is a common step that is missed. Really this should be resolved by updating the build tools that are supplied with the MySQL Server.

Next you will need to follow the configure, make, and install steps:


[brian@zim skeleton-mysql-engine]$ ./configure --with-mysql=/home/brian/mysql/memcache-build-5.1/ --libdir=/home/brian/mysql-builds/mysql-5.1.15/lib/mysql/
[brian@zim skeleton-mysql-engine]$ make
[brian@zim skeleton-mysql-engine]$ make install


The --with-mysql should be pointed to a source tree for MySQL. The --libdir can be used to say which lib directory the plugin should be installed in once it is compiled. As long as you have permission to write to your lib directory, your library will be installed in it when you type "make install"

Now we just need to load it:


mysql> INSTALL PLUGIN skeleton SONAME 'libskeleton_engine.so';
Query OK, 0 rows affected (0.00 sec)

mysql> show plugins;
+------------------+--------+--------------------+-----------------------+---------+
| Name | Status | Type | Library | License |
+------------------+--------+--------------------+-----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| partition | ACTIVE | STORAGE ENGINE | NULL | GPL |
| ARCHIVE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| FEDERATED | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MyISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MRG_MYISAM | ACTIVE | STORAGE ENGINE | NULL | GPL |
| ndbcluster | ACTIVE | STORAGE ENGINE | NULL | GPL |
| SKELETON | ACTIVE | STORAGE ENGINE | libskeleton_engine.so | GPL |
+------------------+--------+--------------------+-----------------------+---------+
12 rows in set (0.06 sec)


You have now loaded your first storage engine!

If you want to try it out, you can do the following:


mysql> create table foo (a varchar(100), b int) ENGINE=SKELETON;
Query OK, 0 rows affected (0.01 sec)

mysql> show create table foo;
+-------+---------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+---------------------------------------------------------------------------------------------------------------------------+
| foo | CREATE TABLE `foo` (
`a` varchar(100) DEFAULT NULL,
`b` int(11) DEFAULT NULL
) ENGINE=SKELETON DEFAULT CHARSET=latin1 |
+-------+---------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


Inserting, updating, and deleting will throw an error at this point. Calling select will result in zero rows.

For the next article we will implement a basic select with no indexes.

Link | Leave a comment | Share

Comments {13}

memcached?

from: tf23
date: May. 5th, 2007 02:46 am (UTC)
Link

>--with-mysql=/home/brian/mysql/memcache-build-5.1/

Can you explain why you're including memcached when you built it?

Reply | Thread

Brian "Krow" Aker

Re: memcached?

from: krow
date: May. 5th, 2007 09:12 am (UTC)
Link

That happens to be the MySQL source tree I am using. No relationship to memcached at all.

Reply | Parent | Thread

Patrick Galbraith

library extensions

from: capttofu
date: May. 7th, 2007 07:58 pm (UTC)
Link

On OSX, my federated odbc engine, when I build it get libraries that have niether .so or .dylib extensions:

my:/usr/local root# ls -l /Users/patg/mysql-build/federated-odbc-engine/src/.libs/
total 3536
-rw-r--r-- 1 patg patg 517824 May 7 15:42 ha_federated_odbc.o
lrwxr-xr-x 1 patg patg 26 May 7 15:42 libfederated_odbc_engine -> libfederated_odbc_engine.0
-rw-r--r-- 1 patg patg 744492 May 7 15:42 libfederated_odbc_engine.0
lrwxr-xr-x 1 patg patg 26 May 7 15:42 libfederated_odbc_engine.0.0.0 -> libfederated_odbc_engine.0
-rw-r--r-- 1 patg patg 525640 May 7 15:42 libfederated_odbc_engine.a
lrwxr-xr-x 1 patg patg 30 May 7 15:42 libfederated_odbc_engine.la -> ../libfederated_odbc_engine.la
-rw-r--r-- 1 patg patg 950 May 7 15:42 libfederated_odbc_engine.lai

On linux, no problem:

patg@ishvara:~/mysql-build/federated-odbc-engine> ls -l src/.libs/
total 948
-rw-r--r-- 1 patg users 345688 2007-05-03 22:31 ha_federated_odbc.o
-rw-r--r-- 1 patg users 343678 2007-05-03 22:31 libfederated_odbc_engine.a
lrwxrwxrwx 1 patg users 30 2007-05-03 22:31 libfederated_odbc_engine.la -> ../libfederated_odbc_engine.la
-rw-r--r-- 1 patg users 976 2007-05-03 22:31 libfederated_odbc_engine.lai
lrwxrwxrwx 1 patg users 33 2007-05-03 22:31 libfederated_odbc_engine.so -> libfederated_odbc_engine.so.0.0.0
lrwxrwxrwx 1 patg users 33 2007-05-03 22:31 libfederated_odbc_engine.so.0 -> libfederated_odbc_engine.so.0.0.0
-rwxr-xr-x 1 patg users 258678 2007-05-03 22:31 libfederated_odbc_engine.so.0.0.0

The "libtool" file in the plugin dir is the same as what mysql-5.1 has in it. So, what else to look at to 'fix' this?

Reply | Thread

got error while compilation

from: nontster
date: Feb. 28th, 2008 03:49 pm (UTC)
Link

mach skeleton_engine-0.7 # ./configure --with-mysql=/home/nont/hpfdb/mysql-5.1.15-beta/ --libdir=/home/nont/hpfdb/

mach skeleton_engine-0.7 # make

then I got this error, any idea ?

Making all in src
make[1]: Entering directory `/home/nont/hpfdb/skeleton_engine-0.7/src'
make all-am
make[2]: Entering directory `/home/nont/hpfdb/skeleton_engine-0.7/src'
/bin/sh ../libtool --preserve-dup-deps --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I/home/nont/hpfdb/mysql-5.1.15-beta//sql -I/home/nont/hpfdb/mysql-5.1.15-beta//include -I/home/nont/hpfdb/mysql-5.1.15-beta//regex -I/home/nont/hpfdb/mysql-5.1.15-beta/ -DMYSQL_DYNAMIC_PLUGIN -g -O2 -fno-implicit-templates -fno-exceptions -fno-rtti -MT libskeleton_engine_la-ha_skeleton.lo -MD -MP -MF .deps/libskeleton_engine_la-ha_skeleton.Tpo -c -o libskeleton_engine_la-ha_skeleton.lo `test -f 'ha_skeleton.cc' || echo './'`ha_skeleton.cc
g++ -DHAVE_CONFIG_H -I. -I/home/nont/hpfdb/mysql-5.1.15-beta//sql -I/home/nont/hpfdb/mysql-5.1.15-beta//include -I/home/nont/hpfdb/mysql-5.1.15-beta//regex -I/home/nont/hpfdb/mysql-5.1.15-beta/ -DMYSQL_DYNAMIC_PLUGIN -g -O2 -fno-implicit-templates -fno-exceptions -fno-rtti -MT libskeleton_engine_la-ha_skeleton.lo -MD -MP -MF .deps/libskeleton_engine_la-ha_skeleton.Tpo -c ha_skeleton.cc -fPIC -DPIC -o .libs/libskeleton_engine_la-ha_skeleton.o
ha_skeleton.cc: In function 'SKELETON_SHARE* get_share(const char*, TABLE*)':
ha_skeleton.cc:102: error: invalid conversion from 'uchar*' to 'const byte*'
ha_skeleton.cc:102: error: initializing argument 2 of 'char* hash_search(const HASH*, const byte*, uint)'
ha_skeleton.cc:118: error: invalid conversion from 'uchar*' to 'const byte*'
ha_skeleton.cc:118: error: initializing argument 2 of 'my_bool my_hash_insert(HASH*, const byte*)'
ha_skeleton.cc:130: error: invalid conversion from 'uchar*' to 'char*'
ha_skeleton.cc:130: error: initializing argument 1 of 'void my_no_flags_free(char*)'
ha_skeleton.cc: In function 'int free_share(SKELETON_SHARE*)':
ha_skeleton.cc:144: error: invalid conversion from 'uchar*' to 'byte*'
ha_skeleton.cc:144: error: initializing argument 2 of 'my_bool hash_delete(HASH*, byte*)'
ha_skeleton.cc:147: error: invalid conversion from 'uchar*' to 'char*'
ha_skeleton.cc:147: error: initializing argument 1 of 'void my_no_flags_free(char*)'
ha_skeleton.cc: In function 'handler* skeleton_create_handler(handlerton*, TABLE_SHARE*, MEM_ROOT*)':
ha_skeleton.cc:158: error: cannot allocate an object of abstract type 'ha_skeleton'
ha_skeleton.h:32: note: because the following virtual functions are pure within 'ha_skeleton':
/home/nont/hpfdb/mysql-5.1.15-beta//sql/handler.h:1235: note: virtual int handler::rnd_next(byte*)
/home/nont/hpfdb/mysql-5.1.15-beta//sql/handler.h:1236: note: virtual int handler::rnd_pos(byte*, byte*)
/home/nont/hpfdb/mysql-5.1.15-beta//sql/handler.h:1249: note: virtual void handler::position(const byte*)
make[2]: *** [libskeleton_engine_la-ha_skeleton.lo] Error 1
make[2]: Leaving directory `/home/nont/hpfdb/skeleton_engine-0.7/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/nont/hpfdb/skeleton_engine-0.7/src'
make: *** [all-recursive] Error 1

Reply | Thread

Brian "Krow" Aker

Re: got error while compilation

from: krow
date: Feb. 28th, 2008 04:00 pm (UTC)
Link

In the beta releases a number of types were changed (and yes that is lame that during a beta we did that).

You need to use a more recent version of MySQL.

Reply | Parent | Thread

Brian "Krow" Aker

Re: got error while compilation

from: krow
date: Feb. 28th, 2008 04:00 pm (UTC)
Link

Oh... by recent I mean recent in the 5.1 version.

Reply | Parent | Thread

Re: got error while compilation

from: nontster
date: Feb. 28th, 2008 04:18 pm (UTC)
Link

thanks krow, I'll try. :)

Reply | Parent | Thread

no luck with mysql-5.1.23-rc

from: nontster
date: Feb. 28th, 2008 04:37 pm (UTC)
Link

I got many errors, which mysql version you had successfully build ?

Reply | Thread

Brian "Krow" Aker

Re: no luck with mysql-5.1.23-rc

from: krow
date: Feb. 29th, 2008 04:54 pm (UTC)
Link

Sorry... I need to pull the code and test. Let me see if I can find the time today.

Reply | Parent | Thread

Re: no luck with mysql-5.1.23-rc

from: nontster
date: Mar. 1st, 2008 05:31 am (UTC)
Link

thanks Krow :)

Reply | Parent | Thread

Re: no luck with mysql-5.1.23-rc

from: nontster
date: Mar. 2nd, 2008 04:46 am (UTC)
Link

Dear krow
I just solve the problem, first mysql source need to run ./configure.
After I run ./configure in mysql and back to skeleton engine the compilation is done properly.

thanks for you help :)

Reply | Parent | Thread

Brian "Krow" Aker

Re: no luck with mysql-5.1.23-rc

from: krow
date: Mar. 2nd, 2008 04:51 am (UTC)
Link

Oh excellent! I will update the readme for skeleton.

Thanks!

Reply | Parent | Thread

No such file or directory error

from: anonymous
date: Feb. 23rd, 2009 08:18 pm (UTC)
Link

First lot of thanks to you for this article.

I have followed the steps stated here. first with mysql source.I did
configure->make->make install
this was all right.
and then
[root@localhost skeleton-mysql-engine]# sh config/bootstrap
upto this all right.Then in mysql
mysql> INSTALL PLUGIN skeleton SONAME 'libskeleton_engine.so';
but this time get an error
ERROR 1126 (HY000): Can't open shared library '/usr/local/mysql-5.1.30/lib/mysql/plugin/libskeleton_engine.so' (errno: 22 cannot open shared object file: No such file or directory)

I failed to find the solution to this problem. I really dont find libskeleton_engine.so file anywhere.
hope to have quick response from you.

Reply | Thread