Rails Active Record .median

Rails neatly provides an .average method to easily find the average value of a column on your db, but it doesn't provide one for finding the median. Thankfully it's really easily to implement

If you just want to add it to one class, put this into the model:

def self.median(column_name)
  median_index = (count / 2)
  # order by the given column and pluck out the value exactly halfway
  order(column_name).offset(median_index).limit(1).pluck(column_name)[0]
end

If you want it to work across all Active Record models in your db... you could add it as a concern (then include it in every class you want it in) or you could just extend ActiveRecord::Base (so it's available to every Active Record, just like .average).

If you want to do that, add the following to /config/initializers/active_record.rb

class ActiveRecord::Base
  def self.median(column_name)
    median_index = (count / 2)
    # order by the given column and pluck out the value exactly halfway
    order(column_name).offset(median_index).limit(1).pluck(column_name)[0]
  end
end

Note: I've tested it even with chained scopes and it seems to work ok on our project (this counts as a final - ie you can't chain after it)

Note: as pointed out to me, the median of an even number of things is the average of the middle two values... and this version doesn't do that - this is a quick-and-dirty version. Feel free to adapt to be mathematically correct if you need that level of precision

Backporting "in_array" to older versions of should'a "ensure_inclusion_of"

Here's another shoulda backport I added recently. If you're still stuck using a legacy Rails system, this backport will let you use "in_array" in the "ensures_inclusion_of" Matcher.

Save it into something like: config/initializers/shoulda_monkeypatches.rb, then use it like this:

   should ensure_inclusion_of(:widget_status).in_array(Widget::VALID_STATUSES).allow_blank.with_message(:is_invalid).use_integer_test_value

  # backport the "in_array" method for the ensure_inclusion_of matcher
  # While we're at it, add allow_blank and allow_nil too
  module Shoulda # :nodoc:
    module ActiveRecord # :nodoc:
      module Matchers
        class EnsureInclusionOfMatcher
          ARBITRARY_OUTSIDE_STRING = 'shouldamatchersteststring'
          ARBITRARY_OUTSIDE_INT = -999999999

          # to initialize the options
          def initialize(attribute)
            super(attribute)
            @options = {}
          end

          # add the method we want to allow us to pass in arrays instead of
          # just ranges
          def in_array(array)
            @array = array
            self
          end          

          # might as well also add the allow_blank and allow_nil methods too
          def allow_blank(allow_blank = true)
            @options[:allow_blank] = allow_blank
            self
          end

          def allow_nil(allow_nil = true)
            @options[:allow_nil] = allow_nil
            self
          end

          # This is a method of my own addition to point out that the
          # test-value must be an Int, not a String... because a string can
          # evaluate to 0 which is a valid Int... which will make the test
          # pass where it shouldn't :P
          def use_integer_test_value(only_integer = true)
            @options[:use_integer_test_value] = only_integer
            self
          end

          # override description so it doesn't just try to inspect the range
          def description
            "ensure inclusion of #{@attribute} in #{inspect_message}"
          end

          # override the matches method to allow arrays as well as ranges
          def matches?(subject)
            super(subject)

            if @range
              @low_message ||= :inclusion
              @high_message ||= :inclusion

              disallows_lower_value &&
                allows_minimum_value &&
                disallows_higher_value &&
                allows_maximum_value
            elsif @array
              if allows_all_values_in_array? && allows_blank_value? && allows_nil_value? && disallows_value_outside_of_array?
                true
              else
                @failure_message_for_should = "#{@array} doesn't match array in validation"
                false
              end
            end
          end

 
        private

          # provide the message-inspect method to use either array or range
          def inspect_message
            @range.nil? ? @array.inspect : @range.inspect
          end          

          # array helper methods
          def allows_all_values_in_array?
            @array.all? do |value|
              allows_value_of(value, @low_message)
            end
          end

          def disallows_value_outside_of_array?
            disallows_value_of(value_outside_of_array)
          end

          def value_outside_of_array
            test_val = @options[:use_integer_test_value] ? ARBITRARY_OUTSIDE_INT : ARBITRARY_OUTSIDE_STRING
            if @array.include?(test_val)
              raise CouldNotDetermineValueOutsideOfArray
            else
              test_val
            end
          end

          # blank and nil helper methods

          def allows_blank_value?
            if @options.key?(:allow_blank)
              blank_values = ['', ' ', "\n", "\r", "\t", "\f"]
              @options[:allow_blank] == blank_values.all? { |value| allows_value_of(value) }
            else
              true
            end
          end

          def allows_nil_value?
            if  @options.key?(:allow_nil)
              @options[:allow_nil] == allows_value_of(nil)
            else
              true
            end
          end         




        end

      end
    end
  end

Making views/triggers/functions work in mysql for rails

We use mysql with Rails. We also use foreign-keys, views, triggers and stored procedures.

Rails's mysql gem still doesn't know how to handle any of the above yet... which is kind of inconvenient if you want to test your application.

The problem arises because even with a SQL schema dump - it only dumps the tables and foreign keys. No views, no triggers, no stored procedures. If you have code that relies on these... your application starts getting MySql errors.

Luckily there seems to be a solution with the db_structure_ext gem, which I've been using happily now for a couple of weeks.

I found the README isn't at all clear as to how to get it set up, so here's some basic instructions on how I made it work with MySQL on Rails 2.3.X

1) Install it as per the README:

gem install db_structure_ext and/or add gem 'db_structure_ext' to Gemfile and then bundle install

2) require it in your Rakefile

# extended db-structure-dump (also does triggers, routines etc)
require 'db_structure_ext/tasks'

3) Monkey-patch ActiveRecord

Create a file call config/initializers/mysql_adapter.rb (or similar) and add the following code:

module ActiveRecord
  module ConnectionAdapters
    class MysqlAdapter
      require 'db_structure_ext/init_mysql_adapter'

      # This is an overridden implementation of the structure_dump so that the
      # rake take db:structure:dump will dump out the schema elements.
      def structure_dump
        connection_proxy = DbStructureExt::MysqlConnectionProxy.new(ActiveRecord::Base.connection)
        connection_proxy.structure_dump
      end

    end
  end
end

You need this so that Active Record actually extends the new methods. If you don't do this, then you can call "structure_dump" yourself in your code, independently for certain tables. But the new functionality won't come through as the default for all of your db tables unless you extend MysqlAdapter as per above.

validate_format_of with better test-names

validate_format_of is almost always called more than once on a single fieldname. Generally with different valid formats... and then with different invalid formats.

eg our set of date-format validations:

  # validate_our_date_format(:end_date, "Please enter correct format for end date")
  def self.validate_our_date_format(field_name,msg)
    # /^[0-9]{1,2}\/[0-9]{1,2}\/[0-9]{4}$/
    # dates we do match and should
    should validate_format_of(field_name).with('01/04/2010').with_message(msg)
    should validate_format_of(field_name).with('31/12/2012').with_message(msg)
    should validate_format_of(field_name).with(' 03/08/2010 ').with_message(msg)
    should validate_format_of(field_name).with('01/02/12').with_message(msg)
    should validate_format_of(field_name).with('02-01-2010').with_message(msg)
    should validate_format_of(field_name).with('1/2/2012').with_message(msg)
    should validate_format_of(field_name).with("04/10/2011\t\n").with_message(msg)
    # dates we don't match and shouldn't
    should validate_format_of(field_name).not_with('42/99/9999').with_message(msg)
    should validate_format_of(field_name).not_with('00/00/0000').with_message(msg)
    should validate_format_of(field_name).not_with('1').with_message(msg)
    should validate_format_of(field_name).not_with('abcd').with_message(msg)
    should validate_format_of(field_name).not_with('01/02/silly').with_message(msg)
  end

but there's a bug in validates_format_of... the description (which becomes the test name) is based on the following code: "{attribute} have a valid format" which makes tests named eg: "should :end_date have a valid format"... These test names are identical, not very useful, and not even properly grammatical.

*ALL* of them are named this eg, in the above example I'd have 12 tests all named as above...

At best, this identical-ness causes scads of Giant Flaming WARNINGS saying that
WARNING: 'test: <your enclosing scope here> should <fieldname> have a valid format.' is already defined
At worst, in certain (old) versions of rails - this means that only the first one gets run at all!

So - uniquifying the name is a Good Thing

The non-usefulness is that even if they do run, you don't know *which* version of your validates_format_of failed... because they're all named the same thing.

Thus this monkey-patch, which puts such things as the test-string *in the description*.

As with my previous shoulda backports, add these to a file in the config/initializers directory

module Shoulda # :nodoc:
  module ActiveRecord
    module Matchers

      # uniquify the description for validates_format_of "should" tests
      class NewValidateFormatOfMatcher < ValidateFormatOfMatcher
        
        # put a little more detail into the description
        def description
          if @value_to_fail
            "not accept #{@value_to_fail} as a valid format for #{@attribute}"
          elsif @value_to_pass
            "accept #{@value_to_pass} as a valid format for #{@attribute}"
          else
            super
          end
        end

      end
      # and override validate_format_of to use our new class
      def validate_format_of(attr)
        NewValidateFormatOfMatcher.new(attr)
      end
    end
  end
end

Note: this monkeypatch is for shoulda version 2.11.1 - not the latest Rails 3 version. You may need to modify accordingly if you're actually up-to-date...

Adding custom HTTP headers to soap4r request

When you're using soap4r, you don't have access to the bare httpclient - which means you have absolutely no way of passing in custom HTTP headers (note, not SOAP headers which definitely have an API).

Monkeypatch time!

See the patchfile below. You need to a) vendor soap4r (if you haven't already), then b) update several files with the given patch.

Then you can pass in your customised headers directly via the soap driver before you call the soap method. Here's an example of use:

driver = SOAP::RPC::Driver.new(url, SCHEMA)
driver.add_method 'MySoapMethod', 'MyParam'

driver.options["protocol.http_custom_headers"] = {'MySpecialHeader1' => 'abc', 'MySpecialHeader2' => 'xyz'}
driver.MySoapMethod 'MyValue'

and here's the necessary patches (note: they're in svn-diff style):

Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (working copy)
@@ -32,6 +32,7 @@
   attr_accessor :allow_unqualified_element
   attr_accessor :default_encodingstyle
   attr_accessor :generate_explicit_type
+  attr_accessor :http_headers
   attr_accessor :use_default_namespace
   attr_accessor :return_response_as_xml
   attr_reader :headerhandler
@@ -57,6 +58,7 @@
     @allow_unqualified_element = true
     @default_encodingstyle = nil
     @generate_explicit_type = true
+    @http_headers = nil
     @use_default_namespace = false
     @return_response_as_xml = false
     @headerhandler = Header::HandlerSet.new
@@ -135,7 +137,9 @@
 
       :generate_explicit_type => @generate_explicit_type,
       :use_default_namespace =>
-        op_info.use_default_namespace || @use_default_namespace
+        op_info.use_default_namespace || @use_default_namespace,
+      :http_headers =>  @http_headers
+
     )
     resopt = create_encoding_opt(
       :envelopenamespace => @options["soap.envelope.responsenamespace"],
@@ -168,6 +172,9 @@
     end
     reqopt[:external_content] = nil
     conn_data = marshal(req_env, reqopt)
+    if reqopt[:http_headers].present?
+      conn_data.extheaders = reqopt[:http_headers]
+    end
     if ext = reqopt[:external_content]
       mime = MIMEMessage.new
       ext.each do |k, v|
@@ -299,6 +305,7 @@
     opt[:default_encodingstyle] = @default_encodingstyle
     opt[:allow_unqualified_element] = @allow_unqualified_element
     opt[:generate_explicit_type] = @generate_explicit_type
+    opt[:http_headers] = @http_headers
     opt[:no_indent] = @options["soap.envelope.no_indent"]
     opt[:use_numeric_character_reference] =
       @options["soap.envelope.use_numeric_character_reference"]
Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (working copy)
@@ -211,6 +211,9 @@
     opt.add_hook("protocol.generate_explicit_type") do |key, value|
       @proxy.generate_explicit_type = value
     end
+    opt.add_hook("protocol.http_custom_headers") do |key, value|
+      @proxy.http_headers = value
+    end
     opt.add_hook("protocol.use_default_namespace") do |key, value|
       @proxy.use_default_namespace = value
     end
Index: vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (working copy)
@@ -34,6 +34,8 @@
     attr_accessor :is_nocontent
     attr_accessor :soapaction
 
+    attr_accessor :extheaders
+
     def initialize(send_string = nil)
       @send_string = send_string
       @send_contenttype = nil
@@ -42,6 +44,7 @@
       @is_fault = false
       @is_nocontent = false
       @soapaction = nil
+      @extheaders = nil
     end
   end
 
@@ -230,6 +233,13 @@
     extheader['Content-Type'] = conn_data.send_contenttype
     extheader['SOAPAction'] = "\"#{ conn_data.soapaction }\""
     extheader['Accept-Encoding'] = 'gzip' if send_accept_encoding_gzip?
+    # allow us to pass in custom HTTP headers
+    if conn_data.extheaders.present?
+      conn_data.extheaders.each do |key, value|
+        extheader[key] = value
+      end
+    end
+
     send_string = conn_data.send_string
     @wiredump_dev << "Wire dump:\n\n" if @wiredump_dev
     begin

Ruby median

Was trawling through the google analytics search keywords and noticed somebody was looking for an Enumerable#median method when they came to this site. Got me thinking...

I went googling for solutions and found that some of them were just plain wrong... can't have that.

module Enumerable
  def median
    # trivial cases
    return nil if self.empty?

    mid = self.length / 2
    if self.length.odd?
      (entries.sort)[mid]
    else
      s = entries.sort
      (s[mid-1] + s[mid]).to_f / 2.0
    end
  end
end

This one isn't the most time-efficient (the sort can get long). I'd need to implement the selection-algorithm for the most optimal solution... but it'll do for a quick fix. I'll post again here if I get around to the better one.

ActiveRecord - find in random order

So we wanted to be able to find a few, random products to put on the homepage of one of our client's sites. I went looking for solutions, and the most commonly spouted one is to do a find with "order by 'rand()'"...

Now, this does the trick, for sure, but the problem is that it will order the entire table by the random amount... and we have a table with many thousand products, and this leads us towards a less-than-optimal query-time - every time somebody hits the homepage. :P

Given we only wanted a handful of products each time - I figured there just had to be a way of pulling out a small set of randomly-selected products without killing the load-time.

then I stumbled upon this article on Random records in rails. It provides a quick-trick fix that will pull out a single random record without all the fuss of ordering the entire table...

...but it doesn't provide a complete, extensible solution. We need it to pull out more than one - and I don't want to have to hit the database multiple times, if I can find a way around it.

Also - we don't just want *any* random product. Some of them are archived or out of stock, and so we need to be able to pass in other finder-option or use our nifty named scopes (eg "in_stock" or "best_sellers").

So here's my new solution. It lets you choose the number of random records to return, and pass in options, and plays nice with named scopes.

 
    # pull out a unique set of random active record objects without killing
    # the db by using "order by rand()"
    # Note: not true-random, but good enough for rough-and-ready use
    #
    # The first param specifies how many you want.
    # You can pass in find-options in the second param
    # examples:
    #  Product.random     => one random product
    #  Product.random(3)  => three random products in random order
    #
    # Note - this method works fine with scopes too! eg:
    #  Product.in_stock.random    => one random product that fits the "in_stock" scope
    #  Product.in_stock.random(3) => three random products that fit the "in_stock" scope
    #  Product.best_seller.in_stock.random => one random product that fits both scopes
    #
    def find_random(num = 1, opts = {})
      # skip out if we don't have any
      return nil if (max = self.count(opts)) == 0

      # don't request more than we have
      num = [max,num].min

      # build up a set of random offsets to go find
      find_ids = [] # this is here for scoping

      # get rid of the trivial cases
      if 1 == num # we only want one - pick one at random
        find_ids = [rand(max)]
      else
        # just randomise the set of possible ids
        find_ids = (0..max-1).to_a.sort_by { rand } 
        # then grab out the number that we need
        find_ids = find_ids.slice(0..num-1) if num != max
      end

      # we've got a random set of ids - now go pull out the records
      find_ids.map {|the_id| first(opts.merge(:offset => the_id)) }
    end

Getting webistrano to deploy under passenger

What's your problem?

I'd been playing with webistrano on my own machine and using it to make deployments to staging - and that was all ticking along nicely. But then it was time to put webistrano up on our staging server - so as to let our sysadmins use it for deployments.

Downloading and installing it is very easy. Configuring it is a little more annoying, but once it was full of all the hosts/systems/rstages/roles and config_options from my existing setup iit shouldn't need any more updating.

Then I tried to deploy. At which point it promptly all fell over.

The same thing happened over and over. I'd click deploy and it'd start. The "running" spinner would spin... forever, the little ajax refresh constantly refreshing the Log... with nothing.
The deploy didn't actually deploy, and what's worse - didn't ever *stop*[1].

I did a deploy and watched top and a ruby process did appear briefly and then disappear - but no ruby process ever showed up in ps...
Not happy Jan :(

I couldn't even cancel the deploy because the cancel button only ever appears once the deployment is fully underway[2]. To deploy again, I had to break locks and do other unsavoury things and that felt really wrong.

So, what was happening and how did we eventually fix it?

Well, the suspicious thing to me was that nothing showed up in the log-section at all. Not even the "loading stage recipe X" that is the very first line of all the successful deploys on my own system.

Thus I figured on a permissions problem. I checked that the runner-user had access to svn, and to the staging server. This was interesting as staging was deploying on itself, we checked that it could ssh into itself happily... and sure enough it asked me to check the 'authenticity of the host' I was connecting to. Now webistrano is notorious for hanging on an un-expected question, so I figured I'd just have to add this to known_hosts and all would be well.

It probably was something that helped... but the deploys were still failing to spin up.

So I dug into the log and found it was chock full of the AJAX Deployment#show refreshes (they're so frequent!) But I eventually got back to the initial Deployment#create which is what should kick off the real deployment. The log for this shows that it goes along fine until right at the bottom, almost completely hidden amongst the noise is one line that rang alarm bells:
sh: ruby: command not found

So I checked permissions again, checked to be sure that ruby was installed, that we could see it in the $PATH as the deployment user, all those things.
I even did a capfile export and ran it with pure capistrano to make sure - and that worked just fine! So now I was really confused.

Finally digging into the webistrano application code, I discovered that the important LOC is in app/models/deployment.rb under def deploy_in_background. It's of the form: system("sh -c \"cd #{RAILS_ROOT} && ruby script/runner -e... etc etc. So I tried this on the command line. ruby -v worked for the deployment user.

I spun up script/console and tried system("sh -c \"ruby -v\"")
and that spat out the correct version and 'true'... so obviously rails could find ruby ok, but running in during deployment was still not happy

Then I copied the above code into the application layout... and it returned false instead of true. Something was happening when inside the running app that wasn't running from the command-line.

Then I discovered this blogpost claiming they also had the log message: sh: ruby command not found

So it seems that when the app is running - by default you're actually not inside a shell - so it's not loading your settings (such as $PATH) and thus not going to find important things (like ruby).

The solution?

Instead of sh -c we need it run under bash --login -c

This will force the process to load up your bash environment settings. The bad news is that you have to monkey-patch webistrano to get it working[3].

Given webistrano is a rails app, this isn't difficult - just annoying. There's only one spot that you need to change. That's the aforementioned deploy_in_background method. Change it there and touch tmp/restart.txt and your deployments should at least spin up now.

anything more?

There is still problem if your recipes also require some $PATH goodness. For example if you are running 'gem bundle' your shell will need to find gem... which means that the recipes need to run under bash too. Now it's a little easier to convince webistrano to do that.

You can override the shell used here by supplying a configuration option: default_shell to bash --login

Note: it's the --login that gets it to do what you want!

Also - don't forget that if you call sh -c explicitly in your recipes you'll need to change it there too.

Notes for webistrano devs:

[1]You should probably surround the deploy process in a timeout.
[2] The cancel button should appear immediately.
[3] It'd be nice if we could configure the shell-command under which webistrano runs

How to monkey patch a gem

Is a gem you're using missing something small - something easy to fix?
You've made a patch and submitted it, but they just aren't responding?

I've found two ways to monkey patch it in place until they get around to pulling your patch in. They both have their pros and cons.

1) vendor the gem in place and hack

Do this if you give up on the gem people ever caring about your change (maybe the gem's been abandoned), if you're only using the gem in a single application; or the patch is only relevant to your one specific application; or if you want to put your changes into the repository for your application.

How:

1. rake gem:unpack into vendor/gems
2. sudo gem uninstall the gem from system if not in use for other apps
3. add the gem code into the repository
4. make your patch in place
5. update the version (eg use jeweller and bump version or hand-hack the gempsec and rename the directory)
6. add a config.gem declaration and depend on your version of the gem OR add a line to your Gemfile - and use vendored_at to point at it

Pros:

1. you can keep track of what changes you've made to the gem in the same place as your application directory - thus it's good for changes that are very specific to your own application-code (ie aren't really relevant or shareable with the wider community or your other apps)
2. it's pretty easy for a quick patch that you know is going to be pulled into the main gem shortly. It's easy to blow away the vendored version once the 'real' version is ready.

Cons:

1. if you're not using gem bundle yet, it's annoying to get your application to use your custom gem
2. it's not easily shareable between your applications if it's hidden in the vendor directory of only one - you may need some complicated extra-directory + symlinking to work...
3. if the gem is ever updated upstream, you have to do nasty things to get the new version (hint: before upgrading, make a patch via your source control... then blow away the gem directory... then download the new gem version... then reapply your patch). :P

2) fork the github repo

If the gem is on github, you can fork the gem there - this is especially good if you're going to reuse your patched gem in multiple applications, and/or make your patches available.

How:

1. Fork the gem OR create a github repo for the gem and push the code up there OR clone locally and create your own repo for it
2. Make your changes, and commit them to the repository as usual
3. In config.gem use :source => 'git::github.org/me/gemname' or gem bundle's Gemfile use :git => 'github.org/me/gemname' (or appropriate location)
4. optionally: be nice and make a pull-request to the original repo

Pros:

1. can easily pull changes from the upstream repository and integrate with your own patches
2. good for sharing amongst multiple of your own applications
3. makes your changes available to other developers that may be facing the same problem
4. good for when the main gem is not longer under development (or only sporadically updated... or has developers that don't agree with your changes)

Cons:

1. more work than a quick hack in vendor
2. must be tracked separately to your own code
3. you might not want to host outside of your own system (of course, nothing wrong with cloning then pushing to your own local repo, rather than github)

Conclusions?

We had a couple of the former and began to run into the issues stated. We discovered, of course, that quick hacks tend to turn into longer, more lasting changes so found that might as well have just done it 'properly' the first time and are now moving towards the latter solution - even setting up our own git-server for the gems we don't want to release back into the wild. YMMV :)

Faking startup and shutdown with ActiveSupport::TestCase

Time-was we didn't just have setup and teardown in our Tests (which are run for every single test case)... we could also have a startup/shutdown pair that ran once per proper Test-case class (ie once at the beginning of all tests in a set).

This was a perfect place to put run-once-only things - especially useful if, say, you wanted to run ruby-prof over your tests. You could put the RubyProf.start in startup and have it print out a callgraph in the shutdown...

ActiveSupport::TestCase doesn't let us have our startup/shutdown anymore... so what to do?

Fake it with at_exit

  # quick-and-dirty class var to tell us if we've already registered the
  # at_exit RubyProf function.
  @@register_done = false
  # register the stop-ruby-prof stuff
  # We want it to spit out a RubyProf callgraph after it's run this test
  # case.
  # You can either: call this function directly in a single test case or
  # test-suite eg: "register_rubyprof"
  # If you pass a value to "RUN_RUBY_PROF" on the command-line - it will
  # also run this register function for you. eg:
  # RUN_RUBY_PROF=true rake test:units TEST=test/unit/widget_test.rb
  def register_rubyprof
    require 'ruby-prof'
    return if @@register_done
    p "starting RubyProf"
    RubyProf.start

    path_base = "#{RAILS_ROOT}/tmp/performance/ruby-prof"

    p "started! Now registering exit-function"
    at_exit do
      p "All done - stopping RubyProf" 
      result = RubyProf.stop

      p "Stopped! now printing call graph"
      timestamp = Time.now.strftime('%Y-%m-%d-%H-%M-%S')
      # try and make the callgraph filename meaningful to the test-run
      filename_base = "rubyprof_#{self.class.name}_#{timestamp}"

      # Print a call tree profile to text for kcachegrind to use
      printer = RubyProf::CallTreePrinter.new(result)
      printer.print(File.open("#{path_base}/#{filename_base}_calltree.txt", 'w'), 0)
      p "All Done. Good luck profiling!"
      end
    @@register_done = true
  end
  # will setup the reguster-rubyprof function if we have set the
  # RUN_RUBYPROF constant to true
  setup :register_rubyprof if RUN_RUBY_PROF

HyRes : ActiveResource that actually works!

It's three months on from my original post whinging about the lack of Validations in Active Resource.

At that time I put my money where my mouth was and forked a copy of the HyperactiveResource plugin, which provided a very crude, basic improvement over vanilla Active Resource.

So What have I achieved?

I've actually done a lot since then. I have implemented a lot of the TODOs that I wrote down as essential for us to get a good, solid basis for a workable system (see the done list below).

I'd class this plugin as working and functionally useful. Right now it's almost as useful as Active Record was a few years back... which is a vast improvement on the basic Active Resource interface.

I think there's still a lot of room for improvement, but the basics are there, and it *feels* like Active Record now - which before it definitely did not. Before it felt like you could play around with it for toy systems, or implement minor bits of functionality - but now you can really implement a fully-functional system on top of it. I know this - because we have done[1].

What's missing?

HyRes still has some niggling issues, and some stuff that would make it a much smoother migration for Active Record. I'm working on these and there is a perpetually-renewed TODO list on the HyRes README file

The funkier aspects of Active Record

I haven't yet re-implemented proper AR reflections - so the associations need some work... you can't do: widget.wodgets.find(...) and you can't use named_scope - which means that many basic plugins (eg restful_authentication) won't work out-of-the-box. But there's enough functionality there that pulling together a system that is totally non-Active Record is feasible.

API assumptions

Currently HyRes makes some assumptions about your API - it assumes a RESTful interface as the optimal configuration. If your API does not match the assumptions, there are some workaround available, but it might not be as useful.

An example is the validates_uniqueness_of method. This currently assumes that your API takes an array of conditions on the query-string, and that this will filter your returned set of findable objects.

If your API doesn't do this... the method currently defaults to fetching *all* objects and filtering on the rails-end... which is likely to be extremely slow (and may lead to timeouts). But it's there in case it's necessary. You may have to simply re-write that method with your own API-requirements in mind. I welcome alternative solutions to the problem...

testing...

Probably one of the nastier downsides atm is that HyRes doesn't have its *own* set of tests... We currently test it through the full suite of tests on our own system. This is mainly due to the fact that it's really quite hard to set up tests that don't rely on an existing API that's up and running. In our system we use a combination of mocks in our functional tests, backed by a fully-functional, running mock API for our integration tests... this is near-impossible for the HyRes plugin itself... so I'm still thinking about how I'll test it independently.

So what *does* it do?

The current list of implemented features (along with how to set up your API to use them) is also available on the HyRes README file but a quick snapshot of the list as of today is below. Note - in some cases, I still find it hard to believe that Active Resource didn't already implement these...

Columns

Because Active Resource isn't backed by a db... you can't use the table columns to determine the known set of attributes of a resource. ARes currently works by accepting any incoming data as an attribute, and using MethodMissing as an accessor for any known attribtues. This is fine for situations where you don't know what attributes will be returned by the remote system.

The downside is that if no value is returned for an attribute and you try to access it... it throws a MethodMissing exception (uuugly!).

If, however, you know what attributes to expect (because it's an agreed API-interface ), it'd be nice to be able to tell Active Resource which attributes we expect - and have it return any missing attributes an nil (rather than explosion).

Thus was born the columns= method.

Currently this method also works a little like attr_accessible - any attributes listed like this will be passed through on create/update... but nothing else will. This allows you to set temporary accessors (eg password and password_confirmation without them being passed over the wire.

Validations

directly pulls them in from ActiveRecord::Validations. For Rails 3.0 - we can use the ActiveModel::Validations component.

Note: validates_uniqueness_of is still experimental - as it requires a network-hit to actually determine whether you have any existing objects with the given field-value. You can't rely on uniqueness-of as there is no way of locking the remote-system's db - and thus somebody could have added a resource with that value in the meantime.

Callbacks

There are now callback hooks for before_validate, before_save etc - all the standard Active Record callbacks we've come to know and love. You can use them for callback chaining a la Active Record (still experimental - may have missed some, but you can definitely use validate :my_validation_method)

Finders

conditional finders

eg Widget.find(:all, :conditions => {:colour => 'blue'}, :limit => 5). This functionality working depends on your API accepting the above filter fields and returning something sensible. There's a lot of doco on how to set up your API (if you have that luxury) in the HyRes README

Dynamic finders and instantiators

eg find_by_X, find_all_by_X, find_by_X, find_last_by_X These rely on your API accepting filter fields as they are really just a bit of a convenient alias for find(:conditions => X). Dynamic finders take any number of arguments using _and_: find_by_X_and_Y_and_Z

We also have dynamic instantiators: find_or_create_by_X OR find_and_instantiate_by_X - both of which also take any number of args, just like the dynamic finders.

Dynamic finders/instantiators also take ! eg: find_or_create_by_X! will throw a ResourceNotValid exception if create fails

Other ActiveRecord::Base-like functions

• update_attribute / update_attributes - actually exist now!
• save! / update_attributes! / update_attribute! / create! - actually exist, now that we can do validation. They raise HyperactiveResource::ResourceNotSaved on failure to save/validate
• ModelName.count (still experimental) - with optional finder-args. This works by first trying a /count request on your model object - and if the route fails it just pulls out all the objects and returns the length (ie - a nasty - but functional fallback).
• updated collection_path that allows suffix_options as well as prefix_options = allows us to generate Rails-style named routes for collection-actions
• no explosion for find/delete_all/destroy_all when none are to be found. (Active Record just returns nil - so should we)
• ActiveRecord-like attributes= (updates rather than replaces)
• ActiveRecord-like load that doesn't dup attributes (stores direct reference)
• reload that does a full clear/fetch to reload (also clears associations cache, now that we have one)

Associations

A lot of the basic associations were in HyRes before I arrived - I've polished a few up (eg adding the belongs_to/has_many functions) and bugfixed

• Resources can be associated with records
• Records can be associated with records
• Awareness of associations between resources: belongs_to, has_many, has_one (note - no HABTM yet)
  • Patient.new.name returns nil instead of MethodMissing
  • Patient.new.races returns [] instead of MethodMissing
  • pat = Patient.new; pat.address_id = 1; pat.address # returns the address object
• Can fetch associations even with a nested route by using the ":nested" option on the nested resource's class. This command automatically adds a prefix-path, and will pre-populate the parent's id when you do an association collection_fetch.
• Supports saving resources that :include other resources via:
  • Nested resource saving (creating a patient will create their associated addresses)
  • Mapping associations ([:address].id will serialize as :address_id)

What's next?

Well, one big thing that's next is that I'm planning on starting to work this stuff back into Rails itself. The brilliant component-architecture and upcoming Active Model changes for Rails 3 are just made for this. What I've been doing on HyRes can feed into that process to make Active Resource what we've always wanted - a real replacement-candidate for Active Model. For this purpose I have a fork of rails - but work will progress slowly (as I'm working on it in my spare time).

Until then, I'll keep on upgrading HyRes as needed for our own corporate requirements. Below, I've listed some features that Active Record provides and that I'd *personally* like to see implemented next... but this is certainly not an exhaustive list. (and I'm very open to other ideas...)

One thing I'd love to see is other people using HyRes for their own projects. I'd love to hear feedback on how it works (or not) as that will feed the project and feed into the work that eventually goes back into Rails. Better-still, if you're willing to work on implementing some of the still-missing features... you would be most welcome. Feel free to check out the Hyperactiveresource project on github and have a go.

Still TODO

1. Testing - inside the plugin! (currently we test via our actual web-app's set of exhaustive tests)
2. MyModel.with_scope/named_scope/default_scope
3. MyModel.find(:include => ...)
4. attr_protected/attr_accessible
5. MyModel.calculate/average/minimum/maximum etc
6. Reflections. There should be no reason why we can't re-use ActiveRecord-style reflections for our associations. They are not SQL-specific. This will also allow a lot more code to automatically Just Work (eg an Active Record could use has_many :through a HyRes)
7. has_and_belongs_to_many
8. Split HyRes into Base and other grouped functions as per AR
9. validates_associated
10. write_attribute that actually hits the remote API ???
11. a default format for when it doesn't understand how to deal with given mime-formats? One which will just pass back the raw data and let you play with it?
12. cache the raw (un-decoded) data onto the object so we don't have to do a second fetch? Or at least allow a universal attribute to be set that turns on caching

Notes

[1] The subject of an upcoming-blogpost will be on the work we've done: "Rewriting monolithic legacy systems in Rails"

Adding prompt to select_tag

Rails's select method is pretty advanced. You can pass in all sorts of funky options to format your selector nicely, including :include_blank => true and the related :prompt => 'Pick one or else!'

select_tag, however, seems to have fallen behind in the usefulness-stakes and implements neither of the above... even though it does an extremely similar thing.

So here's a quick-hack function you can drop into ApplicationHelper to implement that functionality for you.

  # override select_tag to allow the ":include_blank => true" and ":prompt => 'whatever'" options
  include ActionView::Helpers::FormTagHelper
  alias_method :orig_select_tag, :select_tag
  def select_tag(name, select_options, options = {}, html_options = {})
    # remove the options that select_tag doesn't currently recognise
    include_blank = options.has_key?(:include_blank) && options.delete(:include_blank)
    prompt = options.has_key?(:prompt) && options.delete(:prompt)
    # if we didn't pass either - continue on as before
    return orig_select_tag(name, select_options, options.merge(html_options)) unless include_blank || prompt

    # otherwise, add them in ourselves
    prompt_option  = "<option value=\"\">" # to make sure it shows up as nil
    prompt_option += (prompt ? prompt.to_s : "") + "</option>"
    new_select_options = prompt_option + select_options
    orig_select_tag(name, new_select_options, options.merge(html_options))
  end

arrayify!

Following from my post: Array.acts_like_array?, I can now implement arrayify without using is_a? Array or respond_to? :[] && !is_a? String

class Object
  def arrayify
    self.acts_like?(:array) ? self : [self]
  end
end

So we can now do: thing = thing.arrayify or: do_something(thing.arrayify)

Now, what I really want to do is thing.arrayify!, but I'm not sure how to implement that. Any ideas?

Array.acts_like_array?

I've recently discovered the joys of: acts_like?, and was astounded that there hadn't been a simple extension to Array. So here's the monkeypatch:

# extension to allow us to check if an array is an array...
# This feeds the Object.acts_like? method
class Array
  def acts_like_array?
    true
  end
end

Dehumanizing Rails

So we have a string that has been "humanized" - and all the underscores have been stripped out and replaced by spaces, and it's been nicely capitalised (sorry, make that "capitalize"d). So how do we go back the other way?

Plonk the following into config/initializers/inflections.rb (or if you're using a legacy Rails, drop it onto the end of config/environments.rb).

module ActiveSupport::Inflector
  # does the opposite of humanize.... mostly. Basically does a
  # space-substituting .underscore
  def dehumanize(the_string)
    result = the_string.to_s.dup
    result.downcase.gsub(/ +/,'_')
  end
end
class String
  def dehumanize
    ActiveSupport::Inflector.dehumanize(self)
  end
end

Note: it will not reverse any speccy stuff like adding back any "_id"s etc, but it does mean you can do something along the lines of:
assert_equal "network_type", "Network type".dehumanize

Snippet: MyModel.options_for_select

How many times have you written a quickie drop-down box to choose one instance of an associated model? Here's a quickie monkey-patch that will help. Put the following in environment.rb (or similar):

class ActiveRecord::Base
  # convenience method for selection boxes of this model
  # Assumes existance of a function "display_name" that will be a
  # displayable name for the select box.
  def self.options_for_select
    self.find(:all).map {|p| [p.display_name, p.id] }
  end
end

Make sure you have a function in your model named "display_name" that returns what you'd like to see in your drop-down, eg:

class Person < ActiveRecord::Base
  has_many :posts

  # convenience method for the person's name
  def display_name
    [first_name, last_name].compact.join(', ')
  end
end
class Post < ActiveRecord::Base
  belongs_to :person

  alias :display_name :title
end

Then use in a view (eg as follows):

<% form_for(@person) do |f| %>
<p>Pick a post: <%= f.select :post_id, Post.options_for_select -%><%= f.submit 'Go!' -%></p>
<% end %>

Snippet: monkey-patching a gem

Need to make a quick monkey patch to a gem you're using? Follow the steps below:

1. If you don't have a vendor gems directory: svn mkdir vendor/gems
2. cd vendor/gems
3. Unpack the gem into your vendor/gems directory with
   unpack gem <the_gem_name>
4. Checkin the above changes so you have a clean version of the gem to start with
5. Make your monkey patches and check them in
6. Make sure vendor/gems is in the loadpath for gems

And you're done

MySQL hates Timezones

We've started using the TzTime plugin which leverages the Tzinfo gem. It's pretty good. I've adjusted it slightly for my own preferences, but it works pretty well out of the box.

Unfortunately, we ran into some odd issues. Rails suddenly started to find records that it shouldn't. Eg if we were looking for all Event objects that occurred between two given dates, (and expecting to find none) suddenly we were finding one. Which played havoc with our tests :P

The occurred_at date on the Event object was clearly outside of the date-range we were looking at. We even tested to make sure the timezone wasn't stuffing it around and moving it into the date-window.

Eventually we discovered that the to_s(:db) command was printing the date out in a weird format using a T: '2008-07-10T00:50:23+00:00'. This is quite different from the standard MySQL format, which would have it as just: '2008-07-10 00:50:23'

MySQL needs to be set up properly to accept the "T-format" for DateTime objects and ours hasn't been. However, in our case it's much easier to simply extend Rails than to go through the required infrastructure politics to get global database settings changed (which will affect all the other projects in the business).

So I just updated the way that the to_s(:db) format works to always use the 'Y-M-D H:M:S' format as below.

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(:db => '%Y-%m-%d %H:%M:%S')
ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.merge!(:db => '%Y-%m-%d %H:%M:%S')

This seems to work just fine. It means that we'll need to update this if we ever change away from MySQL - but MySQL seems to be hard-coded into this organisation and seems pretty unlikely.

Note: when I first put this code into config/environment.rb it worked fine until I ran the tests - at which point I immediately got an error: uninitialized constant ActiveSupport. This went away the moment I moved the code below the Rails.initializer code (ie underneath the line that says: Include your application configuration below).

Inflector::controllerize

Ok, so I'm bored. After Ben's suggestion for improving POST redirections I decided to implement a "controllerize" inflector ;)

Not sure yet how to get this into String (so I can do: "user".controllerize). I tried going via class String, module Inflections, and via module CoreExt with no joy... so for now you still have to invoke it via Inflector. I welcome suggestions, but I really should stop distracting myself with nifty metaprogramming. ;)

Anyway, this is the code - stuff it into somewhere like environment.rb:

module Inflector
  # turns a given class name into the properly capitlkaised controller name.
  # This can be fed to "constantize" to get the actual controller by this
  # name
  def controllerize(word)
    # make sure it's the raw class name, camelize then add controller 
    "#{word.demodulize.pluralize.camelize}Controller"
  end
end

Examples of use follow:

>> Inflector.controllerize("user")
=> "UsersController"
>> Inflector.controllerize("my_widget")
=> "MyWidgetsController"
>> Inflector.controllerize("String::Thingy")
=> "ThingiesController"
>> ctrl = Inflector.controllerize("user").constantize
=> UsersController
>>

icon buttons

We're all pretty used to the CRUD-based scaffold-ful of actions on a resource index page - such things as "show", "edit" and "delete". But adding a few columns and a few extra actions, causes a swiftly increasing footprint which gets real ugly, real fast.

Buttons stack on top of one another, or they wrap in random ways, or they take up half the width of the screen and squish up all the other data.

The soluion, of course, is to make nifty little icon-buttons. But how do we make this RESTful and DRY?

fixed_button_to

Standard button_to is a wonder - it does practically everything a button needs, and you can override anything...

Well, almost - as I discovered when I first tried to make an image-based button. The "type" is set to submit every time. The following line of code is the culprit:

html_options.merge!("type" => "submit", "value" => name)

So even from the get-go, we need to monkey-patch button_to

You can override the method, or rename it to something else - whatever you like... but sadly you'll need to actually copy the whole button_to code and replace it. So I just created a newly named method. Drop the following into environment.rb (or similar) for buttony joy:

module ActionView::Helpers::UrlHelper
  def fixed_button_to(name, options = {}, html_options = {})
    html_options = html_options.stringify_keys
    convert_boolean_attributes!(html_options, %w( disabled ))

    method_tag = ''
    if (method = html_options.delete('method')) && %w{put delete}.include?(method.to_s)
      method_tag = tag('input', :type => 'hidden', :name => '_method', :value => method.to_s)
    end

    form_method = method.to_s == 'get' ? 'get' : 'post'
   
    request_token_tag = ''
    if form_method == 'post' && respond_to?(:protect_against_forgery?) && protect_against_forgery? 
      request_token_tag = tag(:input, :type => "hidden", :name => request_forgery_protection_token.to_s, 
:value => form_authenticity_token)
    end
    
    if confirm = html_options.delete("confirm")
      html_options["onclick"] = "return #{confirm_javascript_function(confirm)};"
    end

    url = options.is_a?(String) ? options : self.url_for(options)
    name ||= url

    html_options = {"type" => "submit", "value" => name}.merge(html_options)

    "<form method=\"#{form_method}\" action=\"#{escape_once url}\" class=\"button-to\"><div>" +      
method_tag + tag("input", html_options) + request_token_tag + "</div></form>"
  end
end

Those with an unhealthy familiarity with the minutiae of the Rails source will also notice that I added a respond_to?(:protect_against_forgery?) check in there too. My site doesn't use that, and it kept breaking without the check (there must be some magic place its set in a normal Controller/Helper, but it doesn't get in there from here).

icon_button_to

Next up is to write our nifty icon_button_to helper. I use the following:

  # quick and  dodgy icon path-generator
  def get_icon_path(icon_type = 'default')
    icon_name = ACTION_ICONS.include?(icon_type.to_s) ? icon_type : 'default'
    "/images/icons/#{icon_name}.png"
  end

  # generates appropriate image-buttons, given an "icon type" and the usual
  # button options.
  def icon_button_to(icon_type, icon_label, form_path, button_opts = {})
    button_opts.merge!('type' => 'image', :src => get_icon_path(icon_type), 
                       :alt => icon_label, :title => icon_label)
    content_tag 'div', fixed_button_to(icon_label, form_path, button_opts), 
      :class => 'icon_button'
  end

Note that each icon is confined within a div classed icon_button. This allows us to neatly style each button thus:

.icon_button {
  float: left;
  padding: 0 5px 0 0; /* each icon needs a small gap between */
  margin: 0;
}

Iconography

Using the above is simple. It's the same as a normal button_to., but you add the icon_type to the front... and make sure you actually have an icon named after that type sitting in the appropriate image directory. Examples follow:

<%= icon_button_to(:edit, 'Edit order', order_path(order.id), :method => :get) -%>
<%= icon_button_to(:show, 'See order', order_path(order), :method => :get) -%>
<%= icon_button_to(:submit, 'Submit order for processing', submit_order_path(order), 
  :method => :put,  :confirm => 'This will submit the order for processing. Are you sure?') if order.valid_to_submit? -%>
<%= icon_button_to(:delete, 'Delete this order', order_path(order),
  :method => :delete,  :confirm => 'This will delete this order permanently! Are you sure?') unless order.submitted? -%>