Installing Native Gems From Header Files And Shared Libraries

In the previous chapter we learned how to build native gem from sources as well as how to build library that gem depends on. However, sometimes we will have available only header files and shared library for dependent library. Even in this case there is a way to build gem extension from sources, but we must use slightly different approach. Main difference is how we get static library against which we must link gem's extension. Previously we did it by compiling dependent library. In this case we obviously have to use shared library to create link library.

Without going into too much details about windows DLLs let's see what are steps we have to make in order to build gem extension in this case. First one is to create export definition file. This is a file that contains list of all symbols (mainly functions) exported from shared library. Once we have it, in the next step we must create import library for target DLL. Import library is necessary for GCC linker to link gem's extension against dependent shared library.

There are several ways exports definition file can be created. This is ordinary text file, so we can just type in all entries following few simple rules. First line should contain library name, followed by a line with only EXPORTS in it. After these two lines each exported symbol must be listed, one per line. Sample with just a few symbols from a user32.dll, Windows system library, is given below:

LIBRARY USER32.dll
EXPORTS
ActivateKeyboardLayout
AddClipboardFormatListener
AdjustWindowRect
AdjustWindowRectEx
...

It is clear that if we want to create exports definition file we must know, or somehow get, a list of all exported symbols. For large number of symbols this is very slow, boring and error prone task. Fortunately there are tools that can help us automate the whole process. If shared library is MinGW compliant, dlltool, shipped with DevKit, is what we need. Question is what does “MinGW compliant” actually mean?

All shared libraries created with MinGW compliers can be analyzed by dlltool. On the other hand if dll is created with Microsoft Visual Studio, there is a chance dlltool will not be able to create exports definition file from it. In that case we will have to download pexports tool, unpack it to the C:\DevKit\mingw\bin folder and we will be ready to go.

Armed with these tools and knowledge it's now time to use them to build another gem extension. As a sample we'll use mysql2 gem, newer and better version of Ruby binding for MySQL database. Since mysql2 gem is available in binary format it is not likely we will ever have to build it from sources. But in rare cases when bugs are found and fixed before new version is released, having a knowledge how to do it may come in very handy especially because same approach can be used for any other gem.

Mysql2 binding gem requires shared libraries bundled in MySQL installation. If you already installed MySQL you will find required dlls in the lib sub-directory of the root where you've installed database. For the purpose of this book we'll used ZIP archive which you can download from MySQL site. If you look at the lib directory within folder in which you unpacked archive you will see several files with .dll, .lib and .pdb extensions. Files with .dll extension are shared libraries, those with .lib and .pdb are import libraries and debugging symbols created by Microsoft Visual Studio. These import libraries cannot be used by MinGW linker so we'll have to create new ones.

It is already mentioned that shared libraries created with Visual Studio might cause problems for dlltool, but there is no way to know it in advance so let's try to create exports definition file for libmysql.dll. Go to the lib folder where MySQL shared libraries are and execute following command in the Command Prompt:

C:\Ruby\DevKit\devkitvars.bat
cd C:\mysql-5.6.23-win32\lib
C:\mysql-5.6.23-win32\lib>dlltool -z mysql.def –export-all-symbol libmysql.dll

If you now check mysql.def file you will see that symbols list is empty. Apparently dlltool was not able to read exported symbols from shared library. There are number of reasons why dll might not be compatible with dlltool. One cause of incompatibility is difference in sizes of types. For example if long long size is different in GCC and Visual Studio compilers, and struct that has long long members is exported, dlltool will not be able to read information from the shared library.

Dlltool accepts various arguments which you can use to alter the way exports definition file's content is created. You can see them all with dlltool -–help. Above statement tells dlltool to save result in the file mysql.def, to export all symbols found in the shared library and finally passes it the name of the library to be processed. This should be enough for most MinGW compatible dlls.

Since libmysql.dll is not dlltool compatible we have to use different approach and use pexports. While you are in the MySQL's lib directory execute following commands:

C:\mysql-5.6.23-win32\lib>pexports libmysql.dll > mysql.def
C:\mysql-5.6.23-win32\lib>dlltool -d mysql.def -l libmysql.a

First command will create exports definition file and second static library that we will use to link against during gem build. We are now ready to install mysql2 gem from sources. Likewise sqlite3 gem, installing mysql2 from sources will fail because mkmf cannot find required library:

C:\>gem install mysql2 –-platform=ruby
Fetching: mysql2-0.3.18.gem (100%)
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing mysql2:
        ERROR: Failed to build gem native extension.
...

Gem files will remain installed in C:/Ruby/22/lib/ruby/gems/2.2.0/gems/mysql2-0.3.18 for inspection.
Results logged to C:/Ruby/22/lib/ruby/gems/2.2.0/extensions/x86-mingw32/2.2.0/mysql2-0.3.18/gem_make.out

Go to c:\Ruby\22\lib\ruby\gems\2.2.0\gems\mysql2-0.3.18\ext\mysql2\ directory, configure extension and create Makefile with following command:

ruby extconf.rb -- --with-mysql-dir=c:\mysql-5.6.23-win32
checking for ruby/thread.h... yes
checking for rb_thread_call_without_gvl() in ruby/thread.h... yes
checking for rb_thread_blocking_region()... no
checking for rb_wait_for_single_fd()... yes
checking for rb_hash_dup()... yes
checking for rb_intern3()... yes
-----
Using --with-mysql-dir=c:\mysql-5.6.23-win32
-----
checking for mysql.h... yes
checking for errmsg.h... yes
checking for mysqld_error.h... yes
dlltool --kill-at --dllname libmysql.dll --output-lib libmysql.a --input-def C:/Ruby/22/lib/ruby/gems/2.2.0/gems/mysql2-0.3.18/support/libmysql.def
c:/mysql-5.6.23-win32/lib/libmysql.lib
mkdir -p C:/Ruby/22/lib/ruby/gems/2.2.0/gems/mysql2-0.3.18/vendor
cp c:/mysql-5.6.23-win32/lib/libmysql.dll C:/Ruby/22/lib/ruby/gems/2.2.0/gems/mysql2-0.3.18/vendor/libmysql.dll
creating Makefile

Finally build extension library.

c:\Ruby\22\lib\ruby\gems\2.2.0\gems\mysql2-0.3.18\ext\mysql2>make
generating mysql2-i386-mingw32.def
compiling client.c
client.c: In function 'finish_and_mark_inactive':
client.c:585:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
client.c: In function 'rb_mysql_client_query':
client.c:642:7: warning: variable 'async' set but not used [-Wunused-but-set-variable]
client.c: In function 'rb_mysql_client_socket':
client.c:910:3: warning: variable 'wrapper' set but not used [-Wunused-but-set-variable]
compiling infile.c
compiling mysql2_ext.c
compiling result.c
result.c: In function 'msec_char_to_uint':
result.c:186:17: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
result.c: In function 'rb_mysql_result_fetch_fields':
result.c:434:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
linking shared-object mysql2/mysql2.so

All you have to do now is to extract gemspec file like we did with sqlite3 gem and mysql2 gem will be ready to use.