Wednesday 25 November 2009

Convert from config.gem to gem bundler

Why gem bundler?

Our sysadmin hates rake gems:install

It seems to work for me, but all sorts of mayhem ensues when he tries to run it on production... of course I have a sneaking suspicion it's really our fault. After all - we tend to forget that we already happened to globally-install some gem while we were just playing around with it... which means we didn't bother putting it into the config.gem section of environment.rb... oops!

However, there's a new option on the horizon that looks pretty interesting, and is built to sort out some of the uglier issues involved in gem-herding: gem bundler

Yehuda has written a very thorough a tutorial on how to set up gem bundler. But I find it kinda mixes rails and non-rails instructions and it's not so clear on where some things go. I found it a little fiddly to figure out. So here's the step-by step instructions that I used to convert an existing Rails 2.3 project.

Step-by-step instructions

1: Install bundler

sudo gem install bundler

2: create the preinitializer.rb file

Yehuda gave some code in his tutorial that will load everything in the right order. You only need to copy/paste it once and it will then Just Work.

Go grab the code from his tutorial (about halfway down the page) and save it in a file: config/preinitializer.rb

Don't forget to add that file to your source control!

Update: If you're using Passenger, update the code to use:

module Rails
  class Boot
  #...
  end
end

Instead of:

class Rails::Boot
  #...
end

3. create the new gems directory

mkdir vendor/bundler_gems

Add this directory to your source control now - while it's still empty!

While you're at it, open up your source-control's ignore file (eg .gitignore) and add the following:

vendor/bundler_gems/gems
vendor/bundler_gems/specifications
vendor/bundler_gems/doc
vendor/bundler_gems/environment.rb

4. Create the Gemfile

Edit a file called Gemfile in the rails-root of your application (ie right next to your Rakefile)

At the top, add these lines (comments optional):

# because rails is precious about vendor/gems
bundle_path "vendor/bundler_gems"
# this line forces us to use only the bundled gems - making it safer to
# deploy knowing that we won't accidentally assume a gem in existence
# somewhere in the wider world.
disable_system_gems

Again: don't forget to add the Gemfile to your source control!

5. Add your gems to the Gemfile

Now comes the big step - you must copy all your config.gem settings from environment.rb to Gemfile. You can do this almost completely wholesale. For each line, remove the config. from the beginning and then, if they have a version number, remove the :version => and just put the number as the second param. I think an example is in order, so the following line:
config.gem 'rubyist-aasm', :version => '2.1.1', :lib => 'aasm'
becomes:
gem 'rubyist-aasm', '2.1.1', :require => 'aasm'

For most simple gem config lines, this should be enough so that they Just Work. For more complicated config.gem dependencies, refer to the Gemfile documentation in Yehuda's tutorial.

If you already have gems in vendor/gems You can specify that bundler uses them - but you have to be specific about the directory. eg:
gem 'my_cool_gem', '2.1.1', :path => 'vendor/gems/my_cool_gem-2.1.1'

Extra bonus: if you have gems that are *only* important for, say, your test environments, you can add special 'only' and 'except' instructions (or whole sections!) that are environment-specific and keep your staging/production environments gem-free eg:

gem "sqlite3-ruby", :require => "sqlite3" , :only => :development
only :cucumber do
  gem 'cucumber'
  gem 'selenium-client', :require => 'selenium/client'
end
except [:staging, :production] do
  gem 'mocha'          # http://mocha.rubyforge.org/
  gem "redgreen"       # makes your test output pretty!
  gem "rcov"
  gem 'rspec'
end

5a: Add rails

Now... at the top of the Gemfile, add:
gem 'rails', '2.3.4'
(or whatever version you currently use)... otherwise your rails app won't do much! :)

Obviously - if you've vendored rails you will need to specify that in the Gemfile way eg:
gem 'rails', '2.3.4', :path => 'vendor/rails/railties'

If you've opted *not* to disable_system_gems, you won't need this line at all. Alternatively, you could tell the Gemfile to use the system-version anyway thus:
gem 'rails', '2.3.4', :bundle => false

Also, I'd recommend adding the following too:

 gem 'ruby-debug', :except => 'production'  # used by active-support!

6. Let Rails/Rake find your gems:

Edit 'config/environment.rb' and at the bottom (just immediately after the Rails::Initializer.run block, add:

# This is for gem-bundler to find all our gems
require 'vendor/bundler_gems/environment.rb' # add dependenceies to load paths
Bundler.require_env :optional_environment    # actually require the files

7. Give it a whirl

From the rails root directory run gem bundle

The bundler should tell you that it is resolving dependencies, then downloading and installing the gems. You can watch them appearing in the bundler_gems/cache directory :)

and you're done!

...unless something fails and it can't find one - which means you probably forgot to add it to config.gems in the first place! ;)

PS: ok... so I've also noticed you sometimes have to specify gems that your plugins use too - so it may not be entirely your fault... ;)

PPS: if Rake can't find your bundled gems - check that config/preinitializer.rb is set up correctly!

12 comments:

solnic said...

If you're using Passenger you should remember to write:

module Rails
class Boot
...
end
end

in the preinitializer.rb instead of class Rails::Boot like Yehuda suggests because it will fail. Other then that everything works great.

AndrewR said...

>> we tend to forget that we already happened to globally-install some gem while we were just playing around with it

Highly recommend deploying to known-state VM first. Saves embarrasment.

Taryn East said...

@solnic - thanks - yes good point. I haven't copied in the preinitializer code here, but I'll put a note up in the text about that.

@AndrewR - yep - we actually have a staging server for that...but in practice we still sometimes get something that slips through. :P

The worst messes have been where we're changing from a system-installed gem that's already been deployed to a vendored version, or a github fork of a gem we've patched.

So it's technically already there on staging/production, but it's using the wrong one. :P

Taryn East said...

Just so's you know. I'm currently working on how to integrate this nicely with both capistrano, and our Continuous Integration server Hudson... symlinking appropriately etc so we don't have to wholesale checkin all the gems into our source control.

solnic said...

We actually have all the cached gems checked in in our SCM, since a developer has to download them anyway.

BTW - are you happy with Hudson? I want to run away from CC.rb to something else but I'm not sure which alternative I should choose :)

Taryn East said...

@solnic - yep, though I was hoping to not have to check anything into version control, and just maintain a symlinked directory... our deploys are getting bloated enough as it is :P

In any case, I've written up my solution-so-far:
http://rubyglasses.blogspot.com/2009/11/gem-bundle-with-capistrano.html

As to Hudson - seems to be ok. It has it's 'little ways' - and I can't really tell if it's better than CC. I think they both have their own peccadilloes... and I admit I didn't set up Hudson - so I don't know how tough it was to do (and that's often the hurdle I use to figure out usage preferences).

solnic said...

I make symlinks only to gems/gems gems/specifications and gems/environment.rb the cache directory is under SCM.

Re: Hudson, I installed it today and I'm impressed. Much more powerful then CC.rb and it has a nice web UI :)

Taryn East said...

Great stuff! Have you seen the CI game plugin:
http://wiki.hudson-ci.org/display/HUDSON/The+Continuous+Integration+Game+plugin

I can't convince my boss that it's important to install it, but it looks like it'd add a bit of fun... a bit like red and green lava lamps ;)

Kuba Suder said...

I think you have a bug in point 5 - "except [:staging, :production] do" should be "except :staging, :production do" (without the array)...

Taryn East said...

Hi Jakub.
Not sure if that's a bug - it hasn't broken for us. It means we're passing a single parameter that is an array of named environments.

I kinda copied that from the way you do a before_filter exceptions eg:

before_filter :blah, :only => [:new, :create]

That being said, it might be possible to pass it the environments as a separate list of parameters too. :)

Gavin Laking said...

Just a heads up, in the Gemfile, :require_as is now :require

Taryn East said...

Hi Gav, thanks. for the heads-up.