Twitter bootstrap - for quick and easy design

If you're anything like me, for a graphics designer you make a great web developer.

My past attempts at design have run the gamut from "geocities" to "unicorn vomit", and I've become increasingly minimalist in an attempt to rectify that

-> less design == less unicorn-puke == win!

But it still leaves my sites looking like crayon-drawn whitesites

It's a long way from there to the current industry requirements of a clean, flexible, responsive design - able to handle mega-huge desktop-monitors and tiny smart-phones alike while looking sleek and professional... and I had resigned myself to basically just not sucking too badly and hoping my sites were useful enough for people to get past the ugly-duckling look.

but now I've found bootstrap. It lets you quickly and easily build a fully-responsive all-devices-ready design with very little graphic-design nous required. From a full grid-based system (to easily build columns that will nicely collapse on smaller screens), to all sorts of components - navbars, breadcrumbs, panels, lists of linked thumbnail images... the list is pretty huge.

Don't be put off by the "oh my god PURPLE" look of their website - this is a tool worth using.

The getting started guide is where to begin.

There's a number of pre-built templates on bootply templates that even if you don't like, you can cannibalise for the various pieces you want.

You can also buy a whole bunch of pre-packaged, professional themes (for about $12 each) from WrapBootstrap to kickstart your website design. Though be aware they are mostly built on Bootstrap 2

If you're building this stuff in rails, there's a whole bunch of bootsrap-related gems - which mean you don't have to download the css/js versions of bootstrap and mix them all into your assets directory.

Ryan Bates' railscast (see below) uses seyhunak / twitter-bootstrap-rails, which seems to have a whole bunch of neat generators for quickly throwing up layouts. I prefer not mixing bootstrap's Less with the existing SASS, and so have gone for the sass-based gem: bootstrap-sass gem.

I suspect you can combine these two powers for good by using the generators of twitter-bootstrap with the sass-based assets of bootstrap-sass... but have yet to try it (let me know if you do).

There's also a great overview of rails and twitter bootstrap with a related RailsApp that goes into far more depth (I haven't had a look at that yet).

and Ryan Bates has done a railscast on twitter bootstrap basics, and a pro-only railscast: more on twitter bootstrap

enjoy...

---

Note: Bootstrap version 3 has recently been released, and a lot of the tutorials, railscasts etc are based on version 2. Bootply has a Bootstrap 3 migration guide if you want a reference to what has changed.

Making views/triggers/functions work in mysql for rails

We use mysql with Rails. We also use foreign-keys, views, triggers and stored procedures.

Rails's mysql gem still doesn't know how to handle any of the above yet... which is kind of inconvenient if you want to test your application.

The problem arises because even with a SQL schema dump - it only dumps the tables and foreign keys. No views, no triggers, no stored procedures. If you have code that relies on these... your application starts getting MySql errors.

Luckily there seems to be a solution with the db_structure_ext gem, which I've been using happily now for a couple of weeks.

I found the README isn't at all clear as to how to get it set up, so here's some basic instructions on how I made it work with MySQL on Rails 2.3.X

1) Install it as per the README:

gem install db_structure_ext and/or add gem 'db_structure_ext' to Gemfile and then bundle install

2) require it in your Rakefile

# extended db-structure-dump (also does triggers, routines etc)
require 'db_structure_ext/tasks'

3) Monkey-patch ActiveRecord

Create a file call config/initializers/mysql_adapter.rb (or similar) and add the following code:

module ActiveRecord
  module ConnectionAdapters
    class MysqlAdapter
      require 'db_structure_ext/init_mysql_adapter'

      # This is an overridden implementation of the structure_dump so that the
      # rake take db:structure:dump will dump out the schema elements.
      def structure_dump
        connection_proxy = DbStructureExt::MysqlConnectionProxy.new(ActiveRecord::Base.connection)
        connection_proxy.structure_dump
      end

    end
  end
end

You need this so that Active Record actually extends the new methods. If you don't do this, then you can call "structure_dump" yourself in your code, independently for certain tables. But the new functionality won't come through as the default for all of your db tables unless you extend MysqlAdapter as per above.

Acts-as-taggable-on

Tagging is pretty popular these days, and it was time to add it to our site. Unfortunately, lots of the gems are old, and it's had to know if that means "good and has stuck around" or "buggy, obsolete and no longer supported".

The rubytoolbox page on rails tagging has several gems - most of which are marked as inactive now. The only one that looked like it had any recent activity is: acts-as-taggable-on. It's also the top-most-downloaded, so that looked good.

Best news: it is rails-3 compatible AND supports a recent build-version that is still Rails-2 compatible. As I've mentioned before, my current client is still on Rails-2 - because the upgrade pain is not currently outweighed by the new features.

The only annoyance is that the rdoc only has rails-3 post-install instructions rails generate acts_as_taggable_on:migration. These don't work for rails-2, and if you try just substituting "script/" for "rails ", it'll give you an error saying: Couldn't find 'acts_as_taggable_on:migration' generator

I had to hack about a bit to find the new migration name, but what you need is: script/generate acts_as_taggable_on_migration

After that I used this extremely good tutorial on tagging with acts-as-taggable-on.

I don't like the tag-cloud style of tag-selection, and instead prefer something much more like Stack Overflow. So I created my own tag-list as per the code below.

It lists all current tags for the class (assuming similar code setup to the tutorial above), and filter based on that keyword - incorporating any existing search or pagination conditions you already have. It will highlight the current keyword, and change that link to a "deselect if you click" link.

    # code in index page
    <% @tags.sort_by(&:count).reverse.each do |k| %>
      <% url_opts = {:action => "index", :controller => "posts"}
         link_name = "#{k.name} (#{k.count})"
      %>
      <% if @keyword == k.name %>
        <%= link_to link_name, url_opts.merge(:keyword => nil), :class => "tag current_tag", :title => "Click again to see all" %>
      <% else %>
        <%= link_to link_name,  url_opts.merge(:keyword => k.name), :class => "tag", :title => "Click to filter by #{k.name}" %>
      <% end %>
    <% end %>


   # code in controller
   options = {} # any search/pagination conditions go here
   @tags = Post.tag_counts_on(:keywords)
   klass = Post
   klass = klass.tagged_with(@keyword) if (@keyword = params[:keyword]).present?
   @posts = klass.paginate( options )




  /**** and associated tag-cloud styles ****/
  /* basic tag-box */
  .tag {
    background-color: #eee;
    border: 2px solid #ccc;
    color: orange;
    border-radius: 7px;
    -moz-border-radius: 7px;
    padding: 2px 15px;
    text-decoration: none;
  }
  .current_tag {
    background-color: #ddd;
    color: orange;
    border: 2px solid orange;
    border-radius: 7px;
    -moz-border-radius: 7px;
    font-weight: bold;
  }
  .tag:hover, .current_tag:hover {
    background-color: #bbb;
    color: red;
    border: 2px solid red;
    border-radius: 7px;
    -moz-border-radius: 7px;
  }

Next up is to figure out ye olde ajax auto-suggest when I add them.

rubygems upgrade killed rails

Ack! I just tried to upgrade rubygems... and it destroyed my working version of rails 2.3.5 (yes, we use it for a client that has not yet upgraded due to the quite reasonable "it ain't broke" assumption).

Now I can't run script/server without one of the following errors: /usr/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/rails/gem_dependency.rb:268:in `==': undefined method `name' for "Ascii85":String (NoMethodError) or /usr/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/rails/gem_dependency.rb:119:in `requirement': undefined local variable or method `version_requirements' for # (NameError)

Chris Oliver provides a quick fix for getting back to a previously-working version in his description of the undefined local variable or method `version_requirements` and there's an active bug-report for the undefined method `name' for "Ascii85":String

For now, though, it looks like the only solution is to downgrade and hope for a fix... which seems to be coming only for edge rails. I really don't like it that I have to maintain and old version of rubygems just so that I can run my client's perfectly functional rails stack. :(

Adding custom HTTP headers to soap4r request

When you're using soap4r, you don't have access to the bare httpclient - which means you have absolutely no way of passing in custom HTTP headers (note, not SOAP headers which definitely have an API).

Monkeypatch time!

See the patchfile below. You need to a) vendor soap4r (if you haven't already), then b) update several files with the given patch.

Then you can pass in your customised headers directly via the soap driver before you call the soap method. Here's an example of use:

driver = SOAP::RPC::Driver.new(url, SCHEMA)
driver.add_method 'MySoapMethod', 'MyParam'

driver.options["protocol.http_custom_headers"] = {'MySpecialHeader1' => 'abc', 'MySpecialHeader2' => 'xyz'}
driver.MySoapMethod 'MyValue'

and here's the necessary patches (note: they're in svn-diff style):

Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (working copy)
@@ -32,6 +32,7 @@
   attr_accessor :allow_unqualified_element
   attr_accessor :default_encodingstyle
   attr_accessor :generate_explicit_type
+  attr_accessor :http_headers
   attr_accessor :use_default_namespace
   attr_accessor :return_response_as_xml
   attr_reader :headerhandler
@@ -57,6 +58,7 @@
     @allow_unqualified_element = true
     @default_encodingstyle = nil
     @generate_explicit_type = true
+    @http_headers = nil
     @use_default_namespace = false
     @return_response_as_xml = false
     @headerhandler = Header::HandlerSet.new
@@ -135,7 +137,9 @@
 
       :generate_explicit_type => @generate_explicit_type,
       :use_default_namespace =>
-        op_info.use_default_namespace || @use_default_namespace
+        op_info.use_default_namespace || @use_default_namespace,
+      :http_headers =>  @http_headers
+
     )
     resopt = create_encoding_opt(
       :envelopenamespace => @options["soap.envelope.responsenamespace"],
@@ -168,6 +172,9 @@
     end
     reqopt[:external_content] = nil
     conn_data = marshal(req_env, reqopt)
+    if reqopt[:http_headers].present?
+      conn_data.extheaders = reqopt[:http_headers]
+    end
     if ext = reqopt[:external_content]
       mime = MIMEMessage.new
       ext.each do |k, v|
@@ -299,6 +305,7 @@
     opt[:default_encodingstyle] = @default_encodingstyle
     opt[:allow_unqualified_element] = @allow_unqualified_element
     opt[:generate_explicit_type] = @generate_explicit_type
+    opt[:http_headers] = @http_headers
     opt[:no_indent] = @options["soap.envelope.no_indent"]
     opt[:use_numeric_character_reference] =
       @options["soap.envelope.use_numeric_character_reference"]
Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (working copy)
@@ -211,6 +211,9 @@
     opt.add_hook("protocol.generate_explicit_type") do |key, value|
       @proxy.generate_explicit_type = value
     end
+    opt.add_hook("protocol.http_custom_headers") do |key, value|
+      @proxy.http_headers = value
+    end
     opt.add_hook("protocol.use_default_namespace") do |key, value|
       @proxy.use_default_namespace = value
     end
Index: vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (working copy)
@@ -34,6 +34,8 @@
     attr_accessor :is_nocontent
     attr_accessor :soapaction
 
+    attr_accessor :extheaders
+
     def initialize(send_string = nil)
       @send_string = send_string
       @send_contenttype = nil
@@ -42,6 +44,7 @@
       @is_fault = false
       @is_nocontent = false
       @soapaction = nil
+      @extheaders = nil
     end
   end
 
@@ -230,6 +233,13 @@
     extheader['Content-Type'] = conn_data.send_contenttype
     extheader['SOAPAction'] = "\"#{ conn_data.soapaction }\""
     extheader['Accept-Encoding'] = 'gzip' if send_accept_encoding_gzip?
+    # allow us to pass in custom HTTP headers
+    if conn_data.extheaders.present?
+      conn_data.extheaders.each do |key, value|
+        extheader[key] = value
+      end
+    end
+
     send_string = conn_data.send_string
     @wiredump_dev << "Wire dump:\n\n" if @wiredump_dev
     begin

gotcha: Unpacked gem in vendor/gems has no specification file.

Ok, this one had me stumped for a little while. When I was running rake, I'd get these warnings:

  config.gem: Unpacked gem difftmp in vendor/gems has no specification file. Run 'rake gems:refresh_specs' to fix this.
  config.gem: Unpacked gem difftmp in vendor/gems not in a versioned directory. Giving up.

I'd seen this one before - when I'd thoughtlessly unpacked a gem into vendor/gems, then patched it. But for some reason rails doesn't like it when you unpack gems without using rake gems:unpack. I've never been in the habit of using that, because I didn't want to unpack everything. Later, of course I discovered that you can pass "GEM=blah" and it'll only unpack a single gem... but I thought I had done I this time around...

So I blindly followed the instructions to run rake gems:refresh_specs

However, that then spewed on me with an equally-familiar exception message:

rake aborted!
undefined method `installed_source_index' for #<Gem::SourceIndex:0xb76aa218>
... much stacktrace garbage follows...

I sighed deeply and googled the exception... to get the usual suspects, which suggest that I need to refresh the specification file. So I went back to the original error message to see which actual gem was causing the problem.

... then stopped.

You see it became pretty clear that I'd never actually *read* the original error message. "difftmp" is obviously a temporary diff-file showing me the patch I'd made on one of the gems. Now, you could argue that rake shouldn't be stupid and assume that a flat text file is a gem (requiring a specification)... but it was then also quite clear what to do to fix the "bug".

rm vendor/gems/difftmp

Problem solved

Getting webistrano to deploy under passenger

What's your problem?

I'd been playing with webistrano on my own machine and using it to make deployments to staging - and that was all ticking along nicely. But then it was time to put webistrano up on our staging server - so as to let our sysadmins use it for deployments.

Downloading and installing it is very easy. Configuring it is a little more annoying, but once it was full of all the hosts/systems/rstages/roles and config_options from my existing setup iit shouldn't need any more updating.

Then I tried to deploy. At which point it promptly all fell over.

The same thing happened over and over. I'd click deploy and it'd start. The "running" spinner would spin... forever, the little ajax refresh constantly refreshing the Log... with nothing.
The deploy didn't actually deploy, and what's worse - didn't ever *stop*[1].

I did a deploy and watched top and a ruby process did appear briefly and then disappear - but no ruby process ever showed up in ps...
Not happy Jan :(

I couldn't even cancel the deploy because the cancel button only ever appears once the deployment is fully underway[2]. To deploy again, I had to break locks and do other unsavoury things and that felt really wrong.

So, what was happening and how did we eventually fix it?

Well, the suspicious thing to me was that nothing showed up in the log-section at all. Not even the "loading stage recipe X" that is the very first line of all the successful deploys on my own system.

Thus I figured on a permissions problem. I checked that the runner-user had access to svn, and to the staging server. This was interesting as staging was deploying on itself, we checked that it could ssh into itself happily... and sure enough it asked me to check the 'authenticity of the host' I was connecting to. Now webistrano is notorious for hanging on an un-expected question, so I figured I'd just have to add this to known_hosts and all would be well.

It probably was something that helped... but the deploys were still failing to spin up.

So I dug into the log and found it was chock full of the AJAX Deployment#show refreshes (they're so frequent!) But I eventually got back to the initial Deployment#create which is what should kick off the real deployment. The log for this shows that it goes along fine until right at the bottom, almost completely hidden amongst the noise is one line that rang alarm bells:
sh: ruby: command not found

So I checked permissions again, checked to be sure that ruby was installed, that we could see it in the $PATH as the deployment user, all those things.
I even did a capfile export and ran it with pure capistrano to make sure - and that worked just fine! So now I was really confused.

Finally digging into the webistrano application code, I discovered that the important LOC is in app/models/deployment.rb under def deploy_in_background. It's of the form: system("sh -c \"cd #{RAILS_ROOT} && ruby script/runner -e... etc etc. So I tried this on the command line. ruby -v worked for the deployment user.

I spun up script/console and tried system("sh -c \"ruby -v\"")
and that spat out the correct version and 'true'... so obviously rails could find ruby ok, but running in during deployment was still not happy

Then I copied the above code into the application layout... and it returned false instead of true. Something was happening when inside the running app that wasn't running from the command-line.

Then I discovered this blogpost claiming they also had the log message: sh: ruby command not found

So it seems that when the app is running - by default you're actually not inside a shell - so it's not loading your settings (such as $PATH) and thus not going to find important things (like ruby).

The solution?

Instead of sh -c we need it run under bash --login -c

This will force the process to load up your bash environment settings. The bad news is that you have to monkey-patch webistrano to get it working[3].

Given webistrano is a rails app, this isn't difficult - just annoying. There's only one spot that you need to change. That's the aforementioned deploy_in_background method. Change it there and touch tmp/restart.txt and your deployments should at least spin up now.

anything more?

There is still problem if your recipes also require some $PATH goodness. For example if you are running 'gem bundle' your shell will need to find gem... which means that the recipes need to run under bash too. Now it's a little easier to convince webistrano to do that.

You can override the shell used here by supplying a configuration option: default_shell to bash --login

Note: it's the --login that gets it to do what you want!

Also - don't forget that if you call sh -c explicitly in your recipes you'll need to change it there too.

Notes for webistrano devs:

[1]You should probably surround the deploy process in a timeout.
[2] The cancel button should appear immediately.
[3] It'd be nice if we could configure the shell-command under which webistrano runs

How to monkey patch a gem

Is a gem you're using missing something small - something easy to fix?
You've made a patch and submitted it, but they just aren't responding?

I've found two ways to monkey patch it in place until they get around to pulling your patch in. They both have their pros and cons.

1) vendor the gem in place and hack

Do this if you give up on the gem people ever caring about your change (maybe the gem's been abandoned), if you're only using the gem in a single application; or the patch is only relevant to your one specific application; or if you want to put your changes into the repository for your application.

How:

1. rake gem:unpack into vendor/gems
2. sudo gem uninstall the gem from system if not in use for other apps
3. add the gem code into the repository
4. make your patch in place
5. update the version (eg use jeweller and bump version or hand-hack the gempsec and rename the directory)
6. add a config.gem declaration and depend on your version of the gem OR add a line to your Gemfile - and use vendored_at to point at it

Pros:

1. you can keep track of what changes you've made to the gem in the same place as your application directory - thus it's good for changes that are very specific to your own application-code (ie aren't really relevant or shareable with the wider community or your other apps)
2. it's pretty easy for a quick patch that you know is going to be pulled into the main gem shortly. It's easy to blow away the vendored version once the 'real' version is ready.

Cons:

1. if you're not using gem bundle yet, it's annoying to get your application to use your custom gem
2. it's not easily shareable between your applications if it's hidden in the vendor directory of only one - you may need some complicated extra-directory + symlinking to work...
3. if the gem is ever updated upstream, you have to do nasty things to get the new version (hint: before upgrading, make a patch via your source control... then blow away the gem directory... then download the new gem version... then reapply your patch). :P

2) fork the github repo

If the gem is on github, you can fork the gem there - this is especially good if you're going to reuse your patched gem in multiple applications, and/or make your patches available.

How:

1. Fork the gem OR create a github repo for the gem and push the code up there OR clone locally and create your own repo for it
2. Make your changes, and commit them to the repository as usual
3. In config.gem use :source => 'git::github.org/me/gemname' or gem bundle's Gemfile use :git => 'github.org/me/gemname' (or appropriate location)
4. optionally: be nice and make a pull-request to the original repo

Pros:

1. can easily pull changes from the upstream repository and integrate with your own patches
2. good for sharing amongst multiple of your own applications
3. makes your changes available to other developers that may be facing the same problem
4. good for when the main gem is not longer under development (or only sporadically updated... or has developers that don't agree with your changes)

Cons:

1. more work than a quick hack in vendor
2. must be tracked separately to your own code
3. you might not want to host outside of your own system (of course, nothing wrong with cloning then pushing to your own local repo, rather than github)

Conclusions?

We had a couple of the former and began to run into the issues stated. We discovered, of course, that quick hacks tend to turn into longer, more lasting changes so found that might as well have just done it 'properly' the first time and are now moving towards the latter solution - even setting up our own git-server for the gems we don't want to release back into the wild. YMMV :)

gem bundle with capistrano

[Update: There's a capistrano recipe in bundler now, if you just require "bundler/capistrano" apparently it automatically runs bundler install and installs the gems in shared/vendor_bundle]

Ok, so now you've converted your existing rails project to using gem bundle, and it runs fine on your development box, how do you update your capistrano recipes to do it automatically?

Current wisdom is varied. You can:

1. check *everything* into your source control (very heavy!)
2. check in only your Gemfile and the vendor/bundler_gems/cache (which holds all the *.gem files) into source control (still pretty heavy, but makes sure you don't need internet access every time you deploy - use 'gem bundle --cached' in your recipes
3. symlink your vendor/bundler_gems/cache directory to a shared location, and then all you need to checkin is the Gemfile (quicker, still needs to unpack/bundle gems each time)
4. symlink all your vendor/bundler_gems/ directories to a shared location, and then all you need to checkin is the Gemfile (quick and not too heavy)
5. symlink the entire vendor/bundler_gems directory to a shared location (much quicker).

5: Symlink the lot!

This would be my preference. For a single project deployed on production, there should be no reason why we can't just symlink the whole bundled-gem directory, and let the Gemfile keep that directory up-to-date. This feels no different to using a shared file-attachments directory.

Sadly doesn't work due to this:
No such file or directory - MyProjectRailsRoot/vendor/bundler_gems/../../Gemfile
which will resolve to two directories above the *shared symlinked* directory :P

This is because the bundled_gems generates an 'environment.rb' that points back up the directory at the Gemfile that created it... by using a relative path that unfortunately hits the symlink as described above. It'd be nice to be able to tell gem_bundler to make that link absolute...

If anyone knows a way around this, please do let me know!

So we fall back on 3 or 4. My first attempt was to use 3:

3: ci the Gemfile, symlink the cache directory

This seems to be a reasonable compromise. Our deployments are getting pretty damn bloated as it is - with all those plugins and vendored rails etc etc... we don't want to add anything more if we can avoid it. Even gems. We have had no problem with downloading gemfiles, so there's no need to check them in, as long as the deployed directory has access to them when needed. Thus, checking the Gemfile, symlink the cache directory... maybe even symlink, the 'gems' directory sometime too.

So, how to do it?

First - prepare the bundler_gem directory for the first time login to your deployment server, and create the shared directory in, eg:

mkdir ../shared/bundler_gems 
mkdir ../shared/bundler_gems/cache

Next add the symlink and gem bundle commands to your capistrano deployment recipe:

# symlink to the shared gem-cache path then bundle our gems
namespace :gems do
  task :bundle, :roles => :app do
    run "cd #{release_path} && ln -nfs #{shared_path}/bundler_gems/cache #{release_path}/vendor/bundler_gems/cache && gem bundle"
  end
end

after 'deploy:update_code', 'gems:bundle'

And you *should* be able to deploy fine from there. My experience wasn't completely smooth, and I had to set up on the server manually the first time - but from then on the capistrano-task worked fine. Would love to hear your own experiences...

Hudson

If you're using hudson for Continuous integration, you can achieve the same effect by updating the Hudson build script adding:

# prepare gems and shared gem-cache path
ln -nfs <hudson config path>/bundler_gems/cache  vendor/bundler_gems/cache
gem bundle

Also - if you have test/development-environment-only gems... make sure you add your integration-server's environment to the Gemfile environment list, or Hudson won't be able to find, say, rcov or rspec - and then the build will break.

4: ci the Gemfile, symlink the sub-directories

I started with the solution described above, but it still takes a while unpacking all the gems. I'd much prefer to use the solution #5, but that fails due to the relative links in environment.rb. So the ugly compromise is to symlink everything *except* the environment.rb

It works and we can deploy ok with capistrano... but I'm looking for a better solution.

...and after a few deploys...

Well, it was a nice try, but after a few deploys suddenly we started getting breakages... the application couldn't find shoulda for some reason. Now we use a hacked version of shoulda and have vendored it as that's a quickie way to monkey patch a gem.

We told gem bundle where we'd vendored it, and it all seemed to work fine. Unfortunately it broke, and a quick look at the symlinked 'gems' directory tells us why:

rails-2.3.4 -> /var/www/apps/my_app/releases/20091201120443/vendor/rails/railties/
shoulda-2.10.2 -> /var/www/apps/my_app/releases/20091201120443/vendor/bundler_gems/dirs/shoulda

What you can't see are these lines blinking red. What you can see are that these gems are pointing at a specific revision's vendor directory... and not the latest one! Coupled with a :keep_releases = > 4... that release's directory is quite likely to disappear very quickly - and in this case, it already has. This makes these symlinks (set up by gem bundle during release 20091201120443) point to a vendor directory that no longer exists. So it's really not as much of a surprise that our app can't find shoulda anymore.

I think the problem comes along because of gem bundle's rather useful feature of not reloading a gem if it's already installed. It looks to see if the specification exists - if so, it assumes that it's been installed correctly - it doesn't check that the vendored location still exists. That unfortunately spells our doom when capistrano deploys: because gem bundle runs from the newly-created release-directory, that's where the symlink is initially set up. gem bundle doesn't then check it later - even though later its been swept away in a release-cleanup.

So - we're currently working on a fix. Two options present themselves:
1) find a way for gem bundle to symlink from 'releases/current'. This means it has to exist before we do a gem bundle... and that's dangerous because it lets users through into a not-yet-working release. Or
2) we could not vendor any gems - but set up our own gem-server. This will work, but a bit more of a hassle than we prefer for vendored gems.

Troubleshooting:

Git repository 'git://github.com/taryneast/shoulda.git' has not been cloned yet

The recipes online all tell you to use gem bundle --cached and that's a great idea if network connectivity is scarce. As it uses gems that are already in a cache (without going to fetch them remotely)... but it will fail if you don't already have the gems in the cache. SO it relies on you applying solution number 2 above.

There are two solutions:
The first is to just use gem bundle always in your recipes.
The other is to use gem bundle on your development directory - then check the cache into your source control instead of symlinking it to a shared directory. (ie use solution 2 above). This will work just fine, and if you add a new gem, you'll have to make sure you run gem bundle on your working directory before checking in.

I'm not sure of a nice way to get around this if you're using solution 3. It might be worth setting up a rake task that checks if the Gemfile changed and opportunistically runs gem bundle instead of the --cached version (in fact, it'd be nice if gem bundle --cached had a --fetch-if-not-present option!). If you have a solution that worked for you, let me know.

rake aborted! no such file to load -- spec/rake/spectask

I got this when running rcov. It just means you need to add a few extras gems to your Gemfile. We needed all of these. YMMV

  only [:development, :test, :integration] do
    gem 'rcov'  # for test coverage reports
    gem 'rspec' # rcov needs this...
    gem 'ci_reporter' # used somehow by rake+rcov
    gem 'ruby-prof' # used by Hudson for profiling
  end

:bundle => false with disable_system_gems

Looks like these two don't play nice. ie if you choose for a single gem to be unbundled - but have disable_system_gems set - it isn't smart enough to realise that this one gem is meant to be an exception to the 'disable' rule. If you have any unbundled gems, make sure you remove disable_system_gems - or your application simply won't find it.

Convert from config.gem to gem bundler

Why gem bundler?

Our sysadmin hates rake gems:install

It seems to work for me, but all sorts of mayhem ensues when he tries to run it on production... of course I have a sneaking suspicion it's really our fault. After all - we tend to forget that we already happened to globally-install some gem while we were just playing around with it... which means we didn't bother putting it into the config.gem section of environment.rb... oops!

However, there's a new option on the horizon that looks pretty interesting, and is built to sort out some of the uglier issues involved in gem-herding: gem bundler

Yehuda has written a very thorough a tutorial on how to set up gem bundler. But I find it kinda mixes rails and non-rails instructions and it's not so clear on where some things go. I found it a little fiddly to figure out. So here's the step-by step instructions that I used to convert an existing Rails 2.3 project.

Step-by-step instructions

1: Install bundler

sudo gem install bundler

2: create the preinitializer.rb file

Yehuda gave some code in his tutorial that will load everything in the right order. You only need to copy/paste it once and it will then Just Work.

Go grab the code from his tutorial (about halfway down the page) and save it in a file: config/preinitializer.rb

Don't forget to add that file to your source control!

Update: If you're using Passenger, update the code to use:

module Rails
  class Boot
  #...
  end
end

Instead of:

class Rails::Boot
  #...
end

3. create the new gems directory

mkdir vendor/bundler_gems

Add this directory to your source control now - while it's still empty!

While you're at it, open up your source-control's ignore file (eg .gitignore) and add the following:

vendor/bundler_gems/gems
vendor/bundler_gems/specifications
vendor/bundler_gems/doc
vendor/bundler_gems/environment.rb

4. Create the Gemfile

Edit a file called Gemfile in the rails-root of your application (ie right next to your Rakefile)

At the top, add these lines (comments optional):

# because rails is precious about vendor/gems
bundle_path "vendor/bundler_gems"
# this line forces us to use only the bundled gems - making it safer to
# deploy knowing that we won't accidentally assume a gem in existence
# somewhere in the wider world.
disable_system_gems

Again: don't forget to add the Gemfile to your source control!

5. Add your gems to the Gemfile

Now comes the big step - you must copy all your config.gem settings from environment.rb to Gemfile. You can do this almost completely wholesale. For each line, remove the config. from the beginning and then, if they have a version number, remove the :version => and just put the number as the second param. I think an example is in order, so the following line:
config.gem 'rubyist-aasm', :version => '2.1.1', :lib => 'aasm'
becomes:
gem 'rubyist-aasm', '2.1.1', :require => 'aasm'

For most simple gem config lines, this should be enough so that they Just Work. For more complicated config.gem dependencies, refer to the Gemfile documentation in Yehuda's tutorial.

If you already have gems in vendor/gems You can specify that bundler uses them - but you have to be specific about the directory. eg:
gem 'my_cool_gem', '2.1.1', :path => 'vendor/gems/my_cool_gem-2.1.1'

Extra bonus: if you have gems that are *only* important for, say, your test environments, you can add special 'only' and 'except' instructions (or whole sections!) that are environment-specific and keep your staging/production environments gem-free eg:

gem "sqlite3-ruby", :require => "sqlite3" , :only => :development
only :cucumber do
  gem 'cucumber'
  gem 'selenium-client', :require => 'selenium/client'
end
except [:staging, :production] do
  gem 'mocha'          # http://mocha.rubyforge.org/
  gem "redgreen"       # makes your test output pretty!
  gem "rcov"
  gem 'rspec'
end

5a: Add rails

Now... at the top of the Gemfile, add:
gem 'rails', '2.3.4'
(or whatever version you currently use)... otherwise your rails app won't do much! :)

Obviously - if you've vendored rails you will need to specify that in the Gemfile way eg:
gem 'rails', '2.3.4', :path => 'vendor/rails/railties'

If you've opted *not* to disable_system_gems, you won't need this line at all. Alternatively, you could tell the Gemfile to use the system-version anyway thus:
gem 'rails', '2.3.4', :bundle => false

Also, I'd recommend adding the following too:

 gem 'ruby-debug', :except => 'production'  # used by active-support!

6. Let Rails/Rake find your gems:

Edit 'config/environment.rb' and at the bottom (just immediately after the Rails::Initializer.run block, add:

# This is for gem-bundler to find all our gems
require 'vendor/bundler_gems/environment.rb' # add dependenceies to load paths
Bundler.require_env :optional_environment    # actually require the files

7. Give it a whirl

From the rails root directory run gem bundle

The bundler should tell you that it is resolving dependencies, then downloading and installing the gems. You can watch them appearing in the bundler_gems/cache directory :)

and you're done!

...unless something fails and it can't find one - which means you probably forgot to add it to config.gems in the first place! ;)

PS: ok... so I've also noticed you sometimes have to specify gems that your plugins use too - so it may not be entirely your fault... ;)

PPS: if Rake can't find your bundled gems - check that config/preinitializer.rb is set up correctly!

rails gotchas: undefined method `expects'

If you've just installed rails edge and run the tests, they may fail with: NoMethodError: undefined method `expects' for ... etc etc over and over again (along with a few other errors).

Install the mocha gem via rubygems to get rid of a lot of these errors.

It was the line NoMethodError: undefined method `stub' for ... that clued me in.

It seems that rails requires a short list of gems/packages (beyond rails itself) simply to run its own tests... yet there is no rake gem:install task available to help you figure out which ones... and they aren't in the "contributing to rails" guide. I'll be submitting a patch shortly...

Following on from this I got:

MissingSourceFile: no such file to load -- openssl

and farther down the list:

LoadError: no such file to load -- net/https

Unfortunately, these errors occur when Ruby hasn't been installed with openssl-support. If you're running ubuntu, you can just run apt-get install libopenssl-ruby.

soap4r + rails can't find SOAP::Mapping::EncodedRegistry

Joining a new project, I had to go through the usual setting-up rigmarole - which means finding several new things to break/configure/reinstall. One of which was to get soap4r to run.

I'd run gem install soap4r just fine, but running rake was giving me the error below:

rake aborted!
uninitialized constant SOAP::Mapping::EncodedRegistry

Scrounging around gave several solutions that didn't work, mainly using two of the three lines below. I required all three of the following before it ran for me.

# Add these lines to config/environment.rb just before the Rails initializer code
require 'rubygems'
gem 'soap4r'
require 'soap/mapping'

Snippet: monkey-patching a gem

Need to make a quick monkey patch to a gem you're using? Follow the steps below:

1. If you don't have a vendor gems directory: svn mkdir vendor/gems
2. cd vendor/gems
3. Unpack the gem into your vendor/gems directory with
   unpack gem <the_gem_name>
4. Checkin the above changes so you have a clean version of the gem to start with
5. Make your monkey patches and check them in
6. Make sure vendor/gems is in the loadpath for gems

And you're done

Updating gem to 1.2.0 on Ubuntu

Like several others, I tried to gem update today and failed. It gave me weird error telling me it couldn't find the update gem in the repository:

%> sudo gem update --system
Updating RubyGems
Bulk updating Gem source index for: http://gems.rubyforge.org/
Updating rubygems-update
ERROR:  While executing gem ... (Gem::GemNotFoundException)
    could not find rubygems-update locally or in a repository

Googling gave me a fair few other links to people with the same issue. The main one seems to be listed under this ruby-forum topic which started as the release-notes for the Gem update to version 1.2.0. People that had problems updating to this gem listed their steps-to-reproduce in the replies below, and I tried all the suggestions.

sudo gem update --system was the original command I tried and it gave me the error as above. All the fora on the topic insisted that the workaround is to run: gem install rubygems-update -v 1.1.1, but this just gave the same error. I did note that most of these people were on various flavours of Mac, and the pleas for help from those on Ubuntu were drifting away into the sky unheard...

One forum user said he tried the above from his gem directory and that worked for him...
but not for me...

I even tried reinstalling ruby/rubygems via apt-get, but the former was latest version, and the latter doesn't exist...

I was about ready to nuke the site from orbit... and reinstall from scratch and went to the rubyforge page for rubygems. This lists the source code, but I also noticed it had individual "upgrade" gems available for download. This saved me from my previous scorched-earth plan, and eventually resulted in success. For future reference:

1. copy rubygems-update-1.1.0.gem to a local directory
2. then run: sudo gem install rubygems-update-1.1.0.gem
3. repeat with rubygems-update-1.2.0.gem
4. then run update_rubygems

Now it's all fine and sudo gem update --system happily fetches my new gems as before.

Plugins and engines and gems, oh my!

So I was talking to friend of mine about what I was doing and how a gem was too heavy for what I wanted. He pointed out that I should consider doing it as an engine instead of a plugin.

What's the difference between a plugin and an engine you ask?

I didn't know either... engines seem to be the forgotten children of Rails in all the current plugin-love that's going around (not knocking it - I like plugins too).

Anyway, it turns out that an engine is a plugin - with some differences. An engine requires that you install the "engines" plugin first - you can't just script/install the plugin itself. However, adding that engines plugin allows your own baby to do a hell of a lot more than your standard plugin-ly functions.

An engine is a complete, vertical slice of MVC. An ordinary plugin, by comparison, is generally a small chunk of functionality such as a helper library. Engines can have an /app/ directory structure identical to a full rails site. You can also add routes and migrations (with some user-tweaking required). This allows a fully encapsulated chunk of functionality (such as what I was proposing for my little blog).

Now, mephisto and typo are both gems. This is because they contain a huge amount of functionality (they well deserve their leading status in the Rails-blog field). They contain more functionality than what most people associate with a simple plugin... more than what people would associate with an engine (if people used engines more commonly) - so they are distributed as gems.

So, to sum up:

• plugins are for small bundles of helpful code
• engines are for encapsulated slices of vertical functionality and
• gems are for complex, multi-functional systems.

So how do I make an engine?

The information on how to make an engine is a bit scattered about the web. There doesn't seem to be a whole lot out there in the way of tutorials (at least not compared to plugins).

Some of the info seems to be downright incorrect. The article that looked most promising was: the alterthoughts one, but it seems to be based on an old version of engine-construction that Just Doesn't Work. I couldn't get script/generate engine my_blog to give me anything but "can't find a generator for engine". This article is clearly built on an earlier version of both Rails and engines

The railsdoc on engines is fairly complete (though it's not a step-by-step tutorial). It turns out that you can just start out with a rails app - and turn it into an engine. Two things to remember:

Routes

Copy your engine's routes.rb into the root directory. Then instruct your users to add map.from_plugin :your_plugin into their routes.rb.

Migrations

If you have migrations, make sure they are numbered from 001. Then leave instructions for your user to run script/generate plugin_migration, then rake db:migrate.

Shared plugins

A blog needs to share the authentication plugin. But which authentication mechanism are you using? I could assume you're using the same as me (RESTful auth right?) but that's not likely. The best option is to write a wrapper library that allows people to override the functions with their own authentication system, if they choose.

Other plugins

If your plugin depends on other plugins there may be some duplication. I'm sure there must be a way that minimises double-loading of plugins eg if your plugin uses acts_as_taggable, and so does their site - maybe your install.rb could check for that and not bother installing it - but I think that may be over-optimisation.

I'm sure there must be a way to put all these tasks into "install.rb" - but I haven't had a chance to do that yet.

So, what have you done?

So I wrote a very tiny blog app (currently all is does are posts). It was enough to demonstrate the point for me while I figure out how to do the engine thing. I'm still playing with it in my spare time - but I probably won't get in installed until after the Easter long-weekend. At that point I'll also find somewhere to upload it so people can have a play with it.