Installing Native Gems

Now we are ready for Ruby development on Windows. Yet, problem of installing native gems requires a bit more of our attention and better explanation. DevKit's smoke test passed without problems but does it really mean we can install any gem written in C/C++? Unfortunately the answer is no.

Native gems might depend on other libraries which are hard, or sometimes, impossible to build on Windows. Example of such a gem is Nokogiri, HTML and XML parser for Ruby, built upon Libxml2 library. Libxml2 is XML parser and toolkit written in C. If you want to know how to deal, at least partially, with such a libraries you should continue reading this chapter. Otherwise you can freely skip to the section “Installing Ruby On Rails”.

Just for illustration try to install Nokogiri from sources. Installation will fail with quite a long message printed out in the Command Prompt. Leaving out irrelevant parts, output looks similar to following:

c:\>gem install nokogiri --platform=ruby
Fetching: mini_portile-0.6.2.gem (100%)
Successfully installed mini_portile-0.6.2
Fetching: nokogiri-1.6.6.2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing nokogiri:
        ERROR: Failed to build gem native extension.

    C:/Ruby/22/bin/ruby.exe -r ./siteconf20150329-53696-1naoi36.rb extconf.rb
checking if the C compiler accepts ... yes
Building nokogiri using packaged libraries.
************************************************************************
IMPORTANT NOTICE:

Building Nokogiri with a packaged version of zlib-1.2.8.

Team Nokogiri will keep on doing their best to provide security
updates in a timely manner, but if this is a concern for you and want
to use the system library instead; abort this installation process and
reinstall nokogiri as follows:

    gem install nokogiri -- --use-system-libraries
        [--with-xml2-config=/path/to/xml2-config]
        [--with-xslt-config=/path/to/xslt-config]

If you are using Bundler, tell it to use the option:

    bundle config build.nokogiri --use-system-libraries
    bundle install
************************************************************************
...
Extracting libiconv-1.14.tar.gz into tmp/i686-pc-mingw32/ports/libiconv/1.14... OK
Running 'configure' for libiconv 1.14... ERROR, review 'C:/Ruby/22/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.6.2/ext/nokogiri/tmp/i686-pc-mingw32/ports/libiconv/1.14/configure.log' to see what happened.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.
...

Gem files will remain installed in C:/Ruby/22/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.6.2 for inspection.
Results logged to C:/Ruby/22/lib/ruby/gems/2.2.0/extensions/x86-mingw32/2.2.0/nokogiri-1.6.6.2/gem_make.out

Our attempt to install Nokogiri did not succeed due to a missing header file from Libxml2 library. This will happen whenever you try to install native gems that do not have pre-built binaries for Windows system and libraries that these gems depend on are not available.

However this problem is not inherent solely to Windows and Ruby. In order to install C/C++ Ruby extensions on each operating system you must have build tools and all dependencies installed. This means if you do not have Libxml2 installed on the Linux or OS X you will face exactly the same problem. The fact is that most of such gems are developed primarily on Linux where developers, naturally, rely on existing libraries. On the other hand, lot of these libraries are portable and can be used on Windows, but number of them are shipped in a binary format because building them on Windows is quite complicated.

But it is not all that bad as it looks on the first sight. The fact is that at a time of One-Click Ruby Installer which was built with Visual Studio it was very hard to build external libraries and gems. Moreover Visual Studio was commercial and not all of Ruby developers had it. RubyInstaller project has changed a lot in that field.

MRI itself depends on some libraries like libYAML and LibFFI but both of them were built when we were installing Ruby from sources. Moving the whole Ruby build procedure to MSYS/MinGW tool chain was a big step which resulted in possibility to use external libraries built with same compiler tools. This lead to much bigger set of native gems that became available to Ruby developers on Windows.

We will go through the complete procedure of installing one native gem altogether with its external dependency. The gem is Ruby interface for sqlite3 database. First we will try to install gem from sources as we did with Nokogiri.

C:\>gem install sqlite3 --platform=ruby
Fetching: sqlite3-1.3.10.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing sqlite3:
        ERROR: Failed to build gem native extension.

    C:/Ruby/22/bin/ruby.exe extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Install SQLite3 from http://www.sqlite.org/ first.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=C:/Ruby/21/bin/ruby
        --with-sqlite3-dir
        --without-sqlite3-dir
        --with-sqlite3-include
        --without-sqlite3-include=${sqlite3-dir}/include
        --with-sqlite3-lib
        --without-sqlite3-lib=${sqlite3-dir}/lib

extconf failed, exit code 1

Gem files will remain installed in C:/Ruby/22/lib/ruby/gems/2.2.0/gems/sqlite3-1.3.10 for inspection.
Results logged to C:/Ruby/22/lib/ruby/gems/2.2.0/extensions/x86-mingw32/2.2.0/sqlite3-1.3.10/gem_make.out

This time we are displaying complete output of gem install command because we will completely analyze it. We see that script extconf.rb is executed at the beginning of the installation procedure. This is simple program that checks for prerequisites needed for the extension to be built. If all goes well Makefile is created. But something went wrong – C header file sqlite3.h was not found, script has printed out information and exited. If you check content of extconf.rb you will easily find the line responsible for premature script exit.

asplode('sqlite3.h')  unless find_header  'sqlite3.h'

Function find_header, defined in Ruby's mkmf module that creates Makefile, obviously returned false. Question is where it was looking for sqlite3.h file? The answer can be found in mkmf.log file which was created during this installation. Leaving out irrelevant parts here is the answer:

gcc -o conftest.exe -IC:/Ruby/22/include/ruby-2.2.0/i386-mingw32 \
-IC:/Ruby/22/include/ruby-2.2.0/ruby/backward -IC:/Ruby/22/include/ruby-2.2.0 \
-I. -DFD_SETSIZE=2048 -D_WIN32_WINNT=0x0501 -D__MINGW_USE_VC2005_COMPAT \
-D_FILE_OFFSET_BITS=64   -O3 -fno-omit-frame-pointer -fno-fast-math -g -Wall \
-Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long \
-Wno-missing-field-initializers -Wunused-variable -Wpointer-arith \
-Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration \
-Wdeprecated-declarations -Wno-packed-bitfield-compat conftest.c  -L. \
-LC:/Ruby/22/lib -L.      -lmsvcrt-ruby220  -lshell32 -lws2_32 -liphlpapi \
-limagehlp -lshlwapi

We can see several -I... sequences which tell GCC compiler where to search header files. First three directories are related to our Ruby (2.2.1) and the fourth is current directory.

Obviously, in order to compile sqlite3 gem, we first have to get required header file. So let's download tarball containing amalgamation for SQlite 3.8.8.3 with configure script and unpack it to the c:\projects directory where we keep all sources. After extracting, our sqlite3.h file will be in the sqlite-autoconf-3080803 directory and full path to it is exactly what we have to pass to the configuration script.

Next step is to tell compiler where to look for header file. Output of gem installation command can help us here. It displays several configuration options which can be used if we need to alter compilation and linking process. End of list is related to SQlite3 and if you look carefully you will find the one that might help us with missing header file, --with-sqlite3-include. If we pass with this option a full path to a folder where SQlite3 sources are, configuration script will search it too and this might solve our problem.

Fire up new command prompt, activate build tools and go to the folder where Rubygems have left sqlite3 gem's files and start configuration again. Here is needed sequence of commands:

c:\Ruby\DevKit\devkitvars.rb
cd c:\Ruby\22\lib\ruby\gems\2.2.0\gems\sqlite3-1.3.10
ruby setup.rb config -- --with-sqlite3-include=c:\projects\sqlite-autoconf-3080600

Notice two things. We passed config command to the setup.rb script, followed by two dashes surrounded by spaces and configuration option for adding new directory that will be searched for header files. Be careful. Without -- above command will not work. And result is – failure again:

checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... no
....

Configuration script has found header file but now it looks for function libversion_number() in -lsqlite3. Does this ring a bell? And if you substitute -l with lib? You are right! Configuration script tries to find function in the library. Passing libraries which GCC linker should use when it links executable or shared library is done by prefixing library name by -l. Linker will, in our case, search for a libsqlite3 library. If you look in the sqlite-autoconf-3080803 folder you will see that there is no file with that name there. That's because we haven't built SQlite3. We have just downloaded and extracted sources.

Therefore we have to build SQlite3 library. In order do that we must, in the current Command Prompt, start MSYS shell, configure SQlite3 and finally build it. Let's do all this now:

c:\projects\sqlite-autoconf-3080803>c:\Ruby\DevKit\devkitvars.bat
Adding the DevKit to PATH...
c:\projects\sqlite-autoconf-3080803>sh
sh-3.1$./configure
...
c:\projects\sqlite-autoconf-3080803>make

When make command finishes check directory where SQlite3 sources were extracted. You will see new sub-directory .libs with several files in it. The most important to us are libsqlite3.dll.a which is link library used for linking against SQlite3 and libsqlite3-0.dll - shared library used in runtime. Finally we have everything for building sqlite3 gem. Pressing Ctrl-d will close MSYS shell and you will be back in the Windows Command Prompt. Go to the sqlite3 gem folder and run following statements:

ruby setup.rb config -- –-with-sqlite3-lib=C:\projects\sqlite-autoconf-3080600\.libs \
--with-sqlite3-include=c:\projects\sqlite-autoconf-3080600
ruby setup.rb setup

As a result you will get C extension used by this gem. Let's check list of installed gems:

C:\>gem list

*** LOCAL GEMS ***

bigdecimal (1.2.4)
io-console (0.4.2)
...
rdoc (4.1.0)
test-unit (2.1.5.0)

It might be surprising that sqlite3 gem isn't in the list, but what we did is just one step performed by Rubygems during gems installation. We only built C extension and we have to do two more things before sqlite3 gem becomes available for use. We must extract gem specification from the downloaded gem and put it in the specifications directory. Go to Rubygems cache folder where all downloaded gems are saved c:\Ruby\22\lib\ruby\gems\2.2.0\cache and issue following command (you can ignore warnign message):

gem spec sqlite3-1.3.9sq1.gem --ruby > ..\specifications\sqlite3-1.3.9.gemspec
[C:/Ruby/22/lib/ruby/gems/2.2.0/specifications/sqlite3-1.3.9.gemspec] isn't a Gem::Specification (NilClass instead).
gem list

*** LOCAL GEMS ***


bigdecimal (1.2.6)
io-console (0.4.3)
...
rdoc (4.2.0)
sqlite3 (1.3.10)
test-unit (3.0.8)

Finally Rubygems see our newly built sqlite3 gem. But before you check whether gem is working or not, you must copy sqlite3_native.so shared library from c:\Ruby\22\lib\ruby\gems\2.2.0\gems\sqlite3-1.3.10\ext\sqlite3\ folder to c:\Ruby\22\lib\ruby\gems\2.2.0\gems\sqlite3-1.3.10\lib\sqlite3 folder. Now start irb and try to load gem:

C:\>irb
irb(main):001:0> require 'sqlite3'
LoadError: 126: The specified module could not be found.   -  C:/Ruby/22/lib/ruby/gems/2.2.0/gems/sqlite3-1.3.10/lib/sqlite3/sqlite3_native.so
...

Another error! Our shared library sqlite3_native.so is linked against SQlite's shared library, which means that later one will be loaded in runtime. The way Windows searches for shared libraries is cause of this error. When shared library is needed, Windows first checks current application's directory, then system directories (Windows and Windows\System32) and finally all directories listed in the path. SQlite's shared library is not in any of these. We have to put it somewhere in the path so gem's library can find it when needed. Copy it to the the C:\tools folder and try to load gem in irb:

irb(main):002:0> require 'sqlite3'
=> true
irb(main):003:0>

We successfully built not only Ruby gem but also SQlite executable and shared library. After this chapter you should completely understand process of resolving native gems dependencies as well as manually building and installing them.