Rails Active Record .median

Rails neatly provides an .average method to easily find the average value of a column on your db, but it doesn't provide one for finding the median. Thankfully it's really easily to implement

If you just want to add it to one class, put this into the model:

def self.median(column_name)
  median_index = (count / 2)
  # order by the given column and pluck out the value exactly halfway
  order(column_name).offset(median_index).limit(1).pluck(column_name)[0]
end

If you want it to work across all Active Record models in your db... you could add it as a concern (then include it in every class you want it in) or you could just extend ActiveRecord::Base (so it's available to every Active Record, just like .average).

If you want to do that, add the following to /config/initializers/active_record.rb

class ActiveRecord::Base
  def self.median(column_name)
    median_index = (count / 2)
    # order by the given column and pluck out the value exactly halfway
    order(column_name).offset(median_index).limit(1).pluck(column_name)[0]
  end
end

Note: I've tested it even with chained scopes and it seems to work ok on our project (this counts as a final - ie you can't chain after it)

Note: as pointed out to me, the median of an even number of things is the average of the middle two values... and this version doesn't do that - this is a quick-and-dirty version. Feel free to adapt to be mathematically correct if you need that level of precision

ActiveAdmin simple_table

ActiveAdmin by default will build you a table (using table_for) for a collection of objects. But sometimes you just want to throw up a quick table of random values - eg on a dashboard you want a table where the left=column is a description and the right column is, say sets of counts of various things.

ActiveAdmin uses Arbre under the covers to build html for you This means you can use it to laboriously build an html table using all the tags. But if you just want to throw together a super-simple table, this means too much typing.

So I've written a convenience method that will take an array of values and convert it directly to a table.

Note: it does no checking... so make sure your array has the same number of columns for each row

# put this in config/initializers/active_admin.rb
module ActiveAdmin
  module Views
    class SimpleTable < Arbre::HTML::Table
      builder_method :simple_table

      def build(collection)
        table do
         collection.each do |row|
           tr do
             row.each do |col|
               if col.is_a? Numeric
                 td col, style: "text-align: right;"
               else
                 td col
               end
             end
           end
         end
        end
      end
    end
  end
end


# example usage
simple_table [ [ "User count", User.count ],
               [ "Item count", Item.count ],
               [ "Total Wishes", Wish.count ],
               [ "Available Items", Item.available.count ] ]

STOP YELLING! or how to validate a field isn't in ALLCAPS

TuShare lets people give away things they no longer need. It turns out that some people think that putting their headings in ALLCAPS will get more attention... but we think it just annoys people (nobody likes to BE YELLED AT!). Here's a quick snippet that lets you validate that your users aren't YELLING AT YOU! in any given field.

1) create a custom validator in app/validators/not_allcaps_validator.rb with the following code:

# we don't want our users YELLING AT US!
class NotAllcapsValidator < ActiveModel::EachValidator
  def validate_each(record, attribute, value)
    if value.present? && (value.upcase == value)
      record.errors[attribute] << (options[:message] || ""is in ALLCAPS. Please don't YELL AT US (we're sensitive types). Instead, just use normal sentence case. I promise we'll still hear you from over here. :)")
    end
  end
end

2) use it in your model like this:

   validate :name, :description, :not_allcaps => true

3) Add a custom matcher for rspec:

# validates that this field has the "not_allcaps" validator
RSpec::Matchers.define :validate_not_allcaps do |attribute|
  match_for_should do
    match do |model|
      model.send("#{attribute}=", "AN ALLCAPS VALUE")
      !model.valid? && model.errors.has_key?(attribute)
    end
 
    failure_message_for_should do |model|
      "#{model.class} should validate that #{attribute.inspect} cannot accept ALLCAPS VALUES"
    end
 
    failure_message_for_should_not do |model|
      "#{model.class} should validate that #{attribute.inspect} can accept ALLCAPS VALUES"
    end 
  end
end

4) test each field in your model specs with:

    it { should validate_not_allcaps(:name) }
    it { should validate_not_allcaps(:description) }

Making views/triggers/functions work in mysql for rails

We use mysql with Rails. We also use foreign-keys, views, triggers and stored procedures.

Rails's mysql gem still doesn't know how to handle any of the above yet... which is kind of inconvenient if you want to test your application.

The problem arises because even with a SQL schema dump - it only dumps the tables and foreign keys. No views, no triggers, no stored procedures. If you have code that relies on these... your application starts getting MySql errors.

Luckily there seems to be a solution with the db_structure_ext gem, which I've been using happily now for a couple of weeks.

I found the README isn't at all clear as to how to get it set up, so here's some basic instructions on how I made it work with MySQL on Rails 2.3.X

1) Install it as per the README:

gem install db_structure_ext and/or add gem 'db_structure_ext' to Gemfile and then bundle install

2) require it in your Rakefile

# extended db-structure-dump (also does triggers, routines etc)
require 'db_structure_ext/tasks'

3) Monkey-patch ActiveRecord

Create a file call config/initializers/mysql_adapter.rb (or similar) and add the following code:

module ActiveRecord
  module ConnectionAdapters
    class MysqlAdapter
      require 'db_structure_ext/init_mysql_adapter'

      # This is an overridden implementation of the structure_dump so that the
      # rake take db:structure:dump will dump out the schema elements.
      def structure_dump
        connection_proxy = DbStructureExt::MysqlConnectionProxy.new(ActiveRecord::Base.connection)
        connection_proxy.structure_dump
      end

    end
  end
end

You need this so that Active Record actually extends the new methods. If you don't do this, then you can call "structure_dump" yourself in your code, independently for certain tables. But the new functionality won't come through as the default for all of your db tables unless you extend MysqlAdapter as per above.

validate_format_of with better test-names

validate_format_of is almost always called more than once on a single fieldname. Generally with different valid formats... and then with different invalid formats.

eg our set of date-format validations:

  # validate_our_date_format(:end_date, "Please enter correct format for end date")
  def self.validate_our_date_format(field_name,msg)
    # /^[0-9]{1,2}\/[0-9]{1,2}\/[0-9]{4}$/
    # dates we do match and should
    should validate_format_of(field_name).with('01/04/2010').with_message(msg)
    should validate_format_of(field_name).with('31/12/2012').with_message(msg)
    should validate_format_of(field_name).with(' 03/08/2010 ').with_message(msg)
    should validate_format_of(field_name).with('01/02/12').with_message(msg)
    should validate_format_of(field_name).with('02-01-2010').with_message(msg)
    should validate_format_of(field_name).with('1/2/2012').with_message(msg)
    should validate_format_of(field_name).with("04/10/2011\t\n").with_message(msg)
    # dates we don't match and shouldn't
    should validate_format_of(field_name).not_with('42/99/9999').with_message(msg)
    should validate_format_of(field_name).not_with('00/00/0000').with_message(msg)
    should validate_format_of(field_name).not_with('1').with_message(msg)
    should validate_format_of(field_name).not_with('abcd').with_message(msg)
    should validate_format_of(field_name).not_with('01/02/silly').with_message(msg)
  end

but there's a bug in validates_format_of... the description (which becomes the test name) is based on the following code: "{attribute} have a valid format" which makes tests named eg: "should :end_date have a valid format"... These test names are identical, not very useful, and not even properly grammatical.

*ALL* of them are named this eg, in the above example I'd have 12 tests all named as above...

At best, this identical-ness causes scads of Giant Flaming WARNINGS saying that
WARNING: 'test: <your enclosing scope here> should <fieldname> have a valid format.' is already defined
At worst, in certain (old) versions of rails - this means that only the first one gets run at all!

So - uniquifying the name is a Good Thing

The non-usefulness is that even if they do run, you don't know *which* version of your validates_format_of failed... because they're all named the same thing.

Thus this monkey-patch, which puts such things as the test-string *in the description*.

As with my previous shoulda backports, add these to a file in the config/initializers directory

module Shoulda # :nodoc:
  module ActiveRecord
    module Matchers

      # uniquify the description for validates_format_of "should" tests
      class NewValidateFormatOfMatcher < ValidateFormatOfMatcher
        
        # put a little more detail into the description
        def description
          if @value_to_fail
            "not accept #{@value_to_fail} as a valid format for #{@attribute}"
          elsif @value_to_pass
            "accept #{@value_to_pass} as a valid format for #{@attribute}"
          else
            super
          end
        end

      end
      # and override validate_format_of to use our new class
      def validate_format_of(attr)
        NewValidateFormatOfMatcher.new(attr)
      end
    end
  end
end

Note: this monkeypatch is for shoulda version 2.11.1 - not the latest Rails 3 version. You may need to modify accordingly if you're actually up-to-date...

shoulda: ensure_length_of.integer_field and associations with foreign key fields

We're still using Rails 2.3.X and therefore are stuck with thoughtbot-shoulda version 2.11.1 Which unfortunately doesn't come with a lot of the niceties of the most recent version. We've pulled some backports into our initializers directory to bring it up to speed - notably the ability to deal with foreign keys on association-definitions (see below).

But there's one small thing that has always annoyed me about shoulda - and isn't in any version.

That's the ability for ensure_length_of (the validates_length_of macro) to handle integer fields. Active Record can deal with it... therefore shoulda should too.

It's a tiny hack, but I'm currently too lazy to fork shoulda and write the tests to make it a viable pull-request, but here's a monkeypatch that'll get it working for you.

Save it into something like: config/initializers/shoulda_monkeypatches.rb, then use it like this:

   should ensure_length_of(:my_integer).integer_field.is_equal_to(4)

module Shoulda # :nodoc:
  module ActiveRecord
    module Matchers

      # I don't like it that ensure length of only works for strings.
      # This patch makes it also work for integers if we want
      class NewEnsureLengthOfMatcher < EnsureLengthOfMatcher
        def integer_field
          @integer_field = true
          self
        end      

        # re-write "string_of_length" to make the string an int-string
        def string_of_length(length)
          if @integer_field
            '1' * length # yep, this works for an int field
          else
            'x' * length
          end
        end
        
      end      
      # and override ensure_length_of to use our new class
      def ensure_length_of(attr)
        NewEnsureLengthOfMatcher.new(attr)
      end
    end
  end
end

Oh, and for the foreign-key backport:

module Shoulda # :nodoc:
  module Matchers
    module ActiveRecord
      # use the actual foreign key for an association
      class AssociationMatcher
        def foreign_key
          if foreign_key_reflection
            if foreign_key_reflection.respond_to?(:foreign_key)
              foreign_key_reflection.foreign_key.to_s
            else
              foreign_key_reflection.primary_key_name.to_s
            end
          end
        end      
      end      
    end
  end
end

Snippet: Array#to_hash

I've been finding this quite useful - especially when partition just isn't enough:

    def to_hash(&block)
      return {nil => self} unless block_given?
      the_hash = {}
      self.each do |val|
        key = block.call(val)
        the_hash[key] ||= []
        the_hash[key] << val
      end
      the_hash
    end

Usage examples:

   [1,2,3,4,5,6].to_hash {|v| v % 3 }
    => {0=>[3,6], 1=>[1, 4], 2=>[2, 5]}
   Widget.all.to_hash {|w| w.status }
    => {:pending => [<Widget1>,<Widget4>], :complete => [<Widget2>], :awaiting_approval =>[<Widget3>]}

Acts-as-taggable-on

Tagging is pretty popular these days, and it was time to add it to our site. Unfortunately, lots of the gems are old, and it's had to know if that means "good and has stuck around" or "buggy, obsolete and no longer supported".

The rubytoolbox page on rails tagging has several gems - most of which are marked as inactive now. The only one that looked like it had any recent activity is: acts-as-taggable-on. It's also the top-most-downloaded, so that looked good.

Best news: it is rails-3 compatible AND supports a recent build-version that is still Rails-2 compatible. As I've mentioned before, my current client is still on Rails-2 - because the upgrade pain is not currently outweighed by the new features.

The only annoyance is that the rdoc only has rails-3 post-install instructions rails generate acts_as_taggable_on:migration. These don't work for rails-2, and if you try just substituting "script/" for "rails ", it'll give you an error saying: Couldn't find 'acts_as_taggable_on:migration' generator

I had to hack about a bit to find the new migration name, but what you need is: script/generate acts_as_taggable_on_migration

After that I used this extremely good tutorial on tagging with acts-as-taggable-on.

I don't like the tag-cloud style of tag-selection, and instead prefer something much more like Stack Overflow. So I created my own tag-list as per the code below.

It lists all current tags for the class (assuming similar code setup to the tutorial above), and filter based on that keyword - incorporating any existing search or pagination conditions you already have. It will highlight the current keyword, and change that link to a "deselect if you click" link.

    # code in index page
    <% @tags.sort_by(&:count).reverse.each do |k| %>
      <% url_opts = {:action => "index", :controller => "posts"}
         link_name = "#{k.name} (#{k.count})"
      %>
      <% if @keyword == k.name %>
        <%= link_to link_name, url_opts.merge(:keyword => nil), :class => "tag current_tag", :title => "Click again to see all" %>
      <% else %>
        <%= link_to link_name,  url_opts.merge(:keyword => k.name), :class => "tag", :title => "Click to filter by #{k.name}" %>
      <% end %>
    <% end %>


   # code in controller
   options = {} # any search/pagination conditions go here
   @tags = Post.tag_counts_on(:keywords)
   klass = Post
   klass = klass.tagged_with(@keyword) if (@keyword = params[:keyword]).present?
   @posts = klass.paginate( options )




  /**** and associated tag-cloud styles ****/
  /* basic tag-box */
  .tag {
    background-color: #eee;
    border: 2px solid #ccc;
    color: orange;
    border-radius: 7px;
    -moz-border-radius: 7px;
    padding: 2px 15px;
    text-decoration: none;
  }
  .current_tag {
    background-color: #ddd;
    color: orange;
    border: 2px solid orange;
    border-radius: 7px;
    -moz-border-radius: 7px;
    font-weight: bold;
  }
  .tag:hover, .current_tag:hover {
    background-color: #bbb;
    color: red;
    border: 2px solid red;
    border-radius: 7px;
    -moz-border-radius: 7px;
  }

Next up is to figure out ye olde ajax auto-suggest when I add them.

Snippet: random range

Being able to get a random number within a given range is something currently only available in ruby 1.9... but if you want to add a quick method that will do it for you, this one does the trick:

  def rand_in_range(val1,val2)
    rand(1+val2-val1)+val1
  end

  rand_in_range(1,5) # => one of: 1,2,3,4,5

  rand_in_range(100,999) # => anywhere from 100 to 999

Note: it only works for integer values...

Rendering an rjs partial from a controller action

I've recently been working on sharing some AJAXy controller actions between an admin and non-admin controller set up in completely different parts of a site, and this required the ability to use shared rjs templates. Tis is what I found out.

Lets say you originally have an erb and rjs template thus:

   # original update_widget.rjs
   page.replace_html "widget_wrapper", :partial => "widget"

   # original widget.erb
   <div id="widget_wrapper">
      lots of cool stuff here to show off your widget
   </div>

You need to pull that rjs out into a partial template called _update_widget.rjs as below.

Note the full path-name for the partial - this is so we can share this template across controllers

   # new update_widget.rjs
   page << render(:partial => 'widgets/update_widget')

   # new partial: _update_widget.rjs
   page.replace_html "widget_wrapper", :partial => "widgets/widget"

Now to call an rjs template from a controller action you use the syntax as below:

Note: this is what you put in the new controller, the original controller (in this case it'd be widgets_controller) will still default to using the action-name to find the rjs template

   return render( :partial => 'widgets/update_widget.rjs') # the '.rjs' is essential

admin_login_required filter for rubyCAS

Setting up an admin-only login for rubyCAS is fairly straightforward.

First, you have to add an "is_admin" boolean flag to your *local* user class. Remember that you will probably want to restrict admin-rights differently on different applications - so it's better to put it in the local db for this.

If you are really sure that you want universal admins - then you can look into using the "extra attributes" aspects of rubyCAS - but I won't cover this here today.

Now it's just a matter of adding this helper-method to application_controller.rb (note, this depends on logged_in? and current_user)

    # overload the restful_auth version
    def admin_login_required
      # send them to 404 so we don't leak the page's existence
      # Note: implementation of this method left as exercise for the reader...
      render_four_oh_four and return(false) unless logged_in? && @current_user.is_admin?
      true
    end

Now you can add this as a before_filter to whatever actions you need... but make sure it comes *after* the actual login filters eg:

  class MyController < ApplicationController
    # this is the before_filter provided by rubyCAS client
    before_filter CASClient::Frameworks::Rails::Filter
    # this is a call to our own new before_filter which populates the cas user
    before_filter :setup_cas_user

    before_filter :admin_login_required, :only => :admin_homepage

  
    # actions here
    def admin_homepage
      # load cool admin-only stuff here...
    end
  end

To actually update that flag - you'll need to have local versions of user CRUD pages as per any other CRUD pages - affecting the local user values.

This is one article in a series on Rails single-sign-on with rubyCAS

RubyCAS - logged_in?

A logged_in? method is fairly common in rails authentication these days. We use it to determine if there *is* a logged-in user for layouts that are shared public and private - for example to use for greetings, login-only navigation features and displaying the correct login/logout buttons.

Once you've implemented CAS signup and a current_user method, you can use the very simple code below on your ApplicationHelper to get the logged_in? functionality

  def logged_in?
    @current_user.present?
  end

Some authentication applications (eg restful_auth) will use a call to logged_in? as an opportunity to actually try to log a user in... but RubyCAS is more complicated than a simple cookie-login... so I've left actual authentication just to the filter-mechanisms already described.

If you have templates where - if you *can* login a user, you'd like to... but you'd like them also to be available to non-logged in users, then you'll need a Gateway filter.

This is one article in a series on Rails single-sign-on with rubyCAS

Adding custom HTTP headers to soap4r request

When you're using soap4r, you don't have access to the bare httpclient - which means you have absolutely no way of passing in custom HTTP headers (note, not SOAP headers which definitely have an API).

Monkeypatch time!

See the patchfile below. You need to a) vendor soap4r (if you haven't already), then b) update several files with the given patch.

Then you can pass in your customised headers directly via the soap driver before you call the soap method. Here's an example of use:

driver = SOAP::RPC::Driver.new(url, SCHEMA)
driver.add_method 'MySoapMethod', 'MyParam'

driver.options["protocol.http_custom_headers"] = {'MySpecialHeader1' => 'abc', 'MySpecialHeader2' => 'xyz'}
driver.MySoapMethod 'MyValue'

and here's the necessary patches (note: they're in svn-diff style):

Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb (working copy)
@@ -32,6 +32,7 @@
   attr_accessor :allow_unqualified_element
   attr_accessor :default_encodingstyle
   attr_accessor :generate_explicit_type
+  attr_accessor :http_headers
   attr_accessor :use_default_namespace
   attr_accessor :return_response_as_xml
   attr_reader :headerhandler
@@ -57,6 +58,7 @@
     @allow_unqualified_element = true
     @default_encodingstyle = nil
     @generate_explicit_type = true
+    @http_headers = nil
     @use_default_namespace = false
     @return_response_as_xml = false
     @headerhandler = Header::HandlerSet.new
@@ -135,7 +137,9 @@
 
       :generate_explicit_type => @generate_explicit_type,
       :use_default_namespace =>
-        op_info.use_default_namespace || @use_default_namespace
+        op_info.use_default_namespace || @use_default_namespace,
+      :http_headers =>  @http_headers
+
     )
     resopt = create_encoding_opt(
       :envelopenamespace => @options["soap.envelope.responsenamespace"],
@@ -168,6 +172,9 @@
     end
     reqopt[:external_content] = nil
     conn_data = marshal(req_env, reqopt)
+    if reqopt[:http_headers].present?
+      conn_data.extheaders = reqopt[:http_headers]
+    end
     if ext = reqopt[:external_content]
       mime = MIMEMessage.new
       ext.each do |k, v|
@@ -299,6 +305,7 @@
     opt[:default_encodingstyle] = @default_encodingstyle
     opt[:allow_unqualified_element] = @allow_unqualified_element
     opt[:generate_explicit_type] = @generate_explicit_type
+    opt[:http_headers] = @http_headers
     opt[:no_indent] = @options["soap.envelope.no_indent"]
     opt[:use_numeric_character_reference] =
       @options["soap.envelope.use_numeric_character_reference"]
Index: vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb (working copy)
@@ -211,6 +211,9 @@
     opt.add_hook("protocol.generate_explicit_type") do |key, value|
       @proxy.generate_explicit_type = value
     end
+    opt.add_hook("protocol.http_custom_headers") do |key, value|
+      @proxy.http_headers = value
+    end
     opt.add_hook("protocol.use_default_namespace") do |key, value|
       @proxy.use_default_namespace = value
     end
Index: vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb
===================================================================
--- vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (revision 4217)
+++ vendor/gems/soap4r-1.5.8/lib/soap/streamHandler.rb (working copy)
@@ -34,6 +34,8 @@
     attr_accessor :is_nocontent
     attr_accessor :soapaction
 
+    attr_accessor :extheaders
+
     def initialize(send_string = nil)
       @send_string = send_string
       @send_contenttype = nil
@@ -42,6 +44,7 @@
       @is_fault = false
       @is_nocontent = false
       @soapaction = nil
+      @extheaders = nil
     end
   end
 
@@ -230,6 +233,13 @@
     extheader['Content-Type'] = conn_data.send_contenttype
     extheader['SOAPAction'] = "\"#{ conn_data.soapaction }\""
     extheader['Accept-Encoding'] = 'gzip' if send_accept_encoding_gzip?
+    # allow us to pass in custom HTTP headers
+    if conn_data.extheaders.present?
+      conn_data.extheaders.each do |key, value|
+        extheader[key] = value
+      end
+    end
+
     send_string = conn_data.send_string
     @wiredump_dev << "Wire dump:\n\n" if @wiredump_dev
     begin

Ruby median

Was trawling through the google analytics search keywords and noticed somebody was looking for an Enumerable#median method when they came to this site. Got me thinking...

I went googling for solutions and found that some of them were just plain wrong... can't have that.

module Enumerable
  def median
    # trivial cases
    return nil if self.empty?

    mid = self.length / 2
    if self.length.odd?
      (entries.sort)[mid]
    else
      s = entries.sort
      (s[mid-1] + s[mid]).to_f / 2.0
    end
  end
end

This one isn't the most time-efficient (the sort can get long). I'd need to implement the selection-algorithm for the most optimal solution... but it'll do for a quick fix. I'll post again here if I get around to the better one.

Gravatar in rails

Just a quick code-snippet today. A quick helper to take an email and spit out a gravatar image.

  # show the gravatar image for a profile - based on their email
  def gravatar(email, size = 100)
    the_hash = Digest::MD5.hexdigest(email).to_s
    image_tag "http://www.gravatar.com/avatar/#{the_hash}?s=#{size}&d=mm"
  end

This one displays the "mystery man" image if the person doesn't have a gravatar. More details on the options and how to tweak it further are available on the Gravatar Image Requests page.

Profiling inside of procs

I've been using ruby-prof and kcachegrind to delve into our code recently - in an attempt to speed up some of our heaviest views - these are mainly big admin pages listing, say, all orders that are waiting on backordered products.

One of the joys of kcachegrind is the ability to see the actual lines of source-code with the actual time spent in each LOC nicely indicated. You can then click on that line and it'll jump into the slow method that is called by that line.. so you can dig down to where the actual pain is being caused in your app.

One of the problems with with the source-code tracking, though is that it just doesn't deal well with ruby procs. Kcachegrind assumes it's some kind of method-call, but when you click on that LOC, it doesn't take you to the code inside that proc, but you instead end up diving into the *ruby* source-code itself... definitions of blocks and procs etc... which are all aggregated together and thus you can't tease out the bits that you actually want (ie the code operating inside the proc itself).

So if you have something such as the following wrapping almost your entire view page:

   @orders.each do |order|
      # tons of lines....
      # displaying and formatting the order values follow...
      # ...
   end

You'll often only get a single, aggregated line of information n the "orders.each" line... and clicking it only takes you to Ruby's Array#each rather than your actual code.

Entirely unhelpful.

For the sake of profiling that code, though, you can unroll it into a while loop - just to check which LOC are the ones giving the pain by switching it (temporarily) to:

   idx = 0
   while idx < @orders.size do
      order = @orders[idx]
      idx +=1 # don't forget this line or it will take some time to finish.. ;)
      # tons of lines....
      # displaying and formatting the order values follow...
      # ...
   end

It's a dirty hack, but in the absence of "each" it's kind of a necessary evil - and it works. Just make sure to remove it once you're done!!!

Quick RubyProf action filters

A good while back I spoke about using ruby_prof and kcachegrind at LRUG.

I was asked back then if I could stick up my ruby=prof filter-code... and I've been just a *little* bit slack about getting around to that... but recently I've been profiling again, so I thought I'd bring it out again for general use.

Here's a way to quickly add ruby-prof profiling to any given *action*. Use the following around_filter on your action. Note: this will *only* profile what happens in the controller-action. I use this on our uber-heavy summary-page views to make sure that the mammoth Active Record fetches aren't themselves the problem... before getting onto a full-stack profiling-run.

  around_filter :profile_it, :only => :my_slow_action

  # profiling around-filter - just does the controller-action only
  def profile_it
    require 'ruby-prof'

    logger.info "Starting profiling"
    # profile the first set
    RubyProf.start

    # actually do the action
    yield

    result = RubyProf.stop
    logger.info "Profiling finished, printing files"

    # Print the callgraph profile
    filename = Time.now.strftime("RubyProf_#{params[:controller].gsub('/','-')}_#{params[:action].gsub('/','-')}_%Y-%m-%d_%H:%M:%S.grind")
    File.open(filename,'w') do |the_file|
      printer = RubyProf::CallTreePrinter.new(result)
      printer.print(the_file, 0)    
    end
    true
  end

or to profile the full stack through to the template render, you'll want to split the start/stop. I tend to call start_profiling at the top of the controller action (to get rid of most of the rails loading time), and then call stop_profiling at the bottom of the relevant layout.

Put this into application_controller.rb

  # Will only start the profiling... to stop, you'll need to use the helper
  # at the bottom of your template.
  # Use this pair to catch template-rendering in your profiling data
  # Pass ":wall" as a variable to use wall-time instead of process-time
  def start_profiling(time_type = nil)
    require 'ruby-prof'

    logger.info "Starting profiling"

    RubyProf.measure_mode = RubyProf::WALL_TIME if time_type == :wall

    # profile the first set
    RubyProf.start
  end

Put this into application_helper.rb

  # Use this helper in your template to stop profiling. Asumes you have
  # called "start_profiling" in your action somewhere
  # Will asplode if you haven't actually *started* a profiler
  # (eg if the previous action threw an exception before completing the profiling run)
  def stop_profiling
    if RubyProf.running?
      result = RubyProf.stop
      logger.info "Profiling finished, printing files"

      # Print the callgraph profile
      filename = Time.now.strftime("RubyProf_#{params[:controller].gsub('/','-')}_#{params[:action].gsub('/','-')}_%Y-%m-%d_%H:%M:%S.grind")
      File.open(filename,'w') do |the_file|
        printer = RubyProf::CallTreePrinter.new(result)
        printer.print(the_file, 0)    
      end
    else
      raise "Stop profiling called when no profiling started!!!"
    end
    true
  end

Email campaign tracking with Silverpop

SilverPop offer themselves to the world as an "Engagement marketing solution". They provide click-stream and conversion tracking based on your email marketing campaigns, by putting unique identifiers in the links of your mailings (which they can send out in bulk). If you store these IDs and ping them back to SilverPop's click-stream and conversion-tracking servelets, this means you can get complete conversion analysis based on your email marketing campaigns.

I'm currently working for Moneyspyder, and one of our clients has signed up for a Silverpop account - which means we need a way to easily integrate with SilverPop - and the examples they give are all in php (and a little less than dynamic). So here's my Ruby-on-Rails solution.

1) Cookify the IDs

When a customer clicks on a link in one of the SilverPop emails - Silverpop has cleverly adding tracking IDs to identify the mailing-job and individual customer. When we send tracking information back to SilverPop - we need to send back these ids so the click-tracking is stored against the correct campaign and individual.

Of course, the parameters will disappear after the first time the user clicks another link, and we need to persist the data - to keep tracking all the pages they click on until they eventually convert!

The easiest way to keep track of this to stick it all into the session - then we can just check if it's there and send it each time we want to ping to SilverPop... and of course the best place to do session-wrangling is in a before_filter. So stick the following into something like application.rb

  before_filter :save_silverpop_data_in_session

  # This method does the storing of the ids from the URL into a session
  # cookie for later sending.
  # It will replace any existing values in the cookie - which represents the
  # user having returned to the site after viewing (and clicking on) another
  # link from a different email campaign.
  def save_silverpop_data_in_session
    silverpop_params = %w{spmailingid spuserid spjobid spreportid}
    if silverpop_params.all? {|f| params.keys.map{|k|k.to_s.downcase}.include?(f) }
      params_down = {} # ugly but necessary to get past browser case insensitivity issues
      params.each_pair {|k,v| params_down[:"#{k.to_s.downcase}"] = params[k]  if silverpop_params.include?(k.to_s.downcase) }
      session[:silverpop] = {:m => params_down[:spmailingid], :r => params_down[:spuserid],
                             :j => params_down[:spjobid], :rj => params_down[:spreportid]}
    end

  end

2) Configure your pod

SilverPop calls it's servers "pods" - and you could be using any one of them. This works better as a configuration option than hard-coded in your code - and lets you set up a link to the test-server on your dev/test environments. but in environment.rb you'll have something like this:

  # SilverPop URL = mailing list manager/ClickStream Analysis
  SILVERPOP_SITE_URL =  "recp.mkt41.net"
  SILVERPOP_SITE_HTTPS_URL =  "marketer4.silverpop.com"

3) Helping hands

Next up is to figure out how to ping SilverPop... which they helpfully give us an img tag example of how to build up the correct url with all the requisite values.

SilverPop has two servelets - one for accepting pings for click-stream tracking and one for the conversions. But they're almost identical, just taking different parameters... and I'm lazy and don't want to have to remember all the common details of how to do this in each place in the site. Thus: helpers to the rescue!

  # This method generates the code that calls the ClickStream tracking
  # servelet - we pass in the page name and page URL, along with the values
  # passed through from the last silverpop email - saved in the session
  def silverpop_click_stream_ping(page_name, page_url)
    silverpop_link('cst', :name => page_name, :s => page_url)
  end


  # This method generates the code that calls the Conversion Tracking
  # servelet - we pass in the "action", "detail" and "value" flags - 
  # eg "silverpop_conversion_ping 'CompletedOrder', order.id, order.grand_total" 
  # along with the values passed through from the last silverpop email -
  # saved in the session
  def silverpop_conversion_ping(action,detail,value = nil)
    silverpop_link('cot', :a => action, :d => detail, :amt => value)
  end


  # generate the SilverPop ping-image based on required servelet and parameters
  def silverpop_link(servelet, options)
    # skip out early if this user hasn't come through a silverpop email
    return nil unless session[:silverpop].present?

    base_url = (/https/ =~ request.protocol) ?  "https://#{SILVERPOP_SITE_HTTPS_URL}" : "http://#{SILVERPOP_SITE_URL}" 

    image_tag "#{base_url}/#{servelet}?#{session[:silverpop].merge(options).to_query}", 
      :height => 1, :width => 1, :alt => ""
    # we'd like this alt-tag: "Silverpop #{servelet.upcase} Servelet Ping"
    # to be accessible... but the above is not *really* an image, and thus will display 
    # the alt-text in the html at present... this sux, but I have yet to talk to silverpop 
    # about a way around this.
  end

4) Click-stream - analyse!

So, now it's time to get down and dirty with the click-stream analysis. This couldn't be more simple. Just use the helper to pop a link into your main layout. eg:

  <!-- silverpop cst ping -->>
  <%= silverpop_click_stream_ping(@page_title, url_for(:only_path => false)) -%>

Now, as you can see, this mainly works by using our dynamic page-title and a link to the current-page (using the empty url_for trick). Note that if you leave off the "only_path=false" option it won't provide the hostname. This may be what you want if you want to roll up multiple mirrored domains. You'll also have to adjust the page-name parameter as necessary for the way you generate the page-title... but otherwise you're good to go and this means every page-click gets tracked back to SilverPop from now on.

With one caveat... if you have AJAX-updating, you may need to figure out a neato trick of putting the ping-link into the newly-generated page-pieces or these "clicks" won't get tracked as the layout won't see them as new pages. I'll leave that as an exercise for the reader as it's very site-specific.

5) Mine your conversion gold

Now only the final, and most important, part is left - tracking actual conversions.

A lot has already been written about what constitutes an important conversion for your site. I won't repeat it all here as you can track heaps, and it's really down to what is important for your business. So I'll pretend we only care about when a customer completes an order - which we know because they land on the "thank you" page.

Which means we need to pop a link to the COT servelet there and pass in the important details... nothing easier:

  <!-- silverpop cot ping -->>
  <%= silverpop_conversion_ping("Order Complete", @order.id, @order.grand_total) -%>

Now SilverPop will monitor our marketing mails from go to whoa - and even know which order they completed and, most important, how much money they ended up giving us... Gold!

Update: Sorry for the repost (anyone that noticed). I fixed a few bugs after I finally got a chance to fully test this out.

Anybody implementing a Silverpop solution should also be aware that Silverpop's servers (at this moment) do not correctly redirect links from mailings sent from their test server. Instead you get redirected to your website, but without any values in the required query parameters (eg spMailingID etc are empty).

To do an end-to-end test of SilverPop's link redirection, you must create a real newsletter mailing on the live server, and just restrict the recipients (eg send it to yourself and any other devs on your team).

Stored Procedure-fu

So, I've been tuning my db performance on our heaviest database call... and this involved adding a quickie stored procedure to make life a little simpler.

It's nothing complex, just does a really simple distance calculation (using pythagorus)... which we needed because most versions of MySql don't come fully-loaded with all those nice functions that you can find in the manual.

Now the problem with stored procedures, is how to add them to the db without too much hassle. I have heard there are gems out there - but they seem a little like overkill, and the ones I've seen have been "build it on the fly"... I just want to put it into the db and leave it there.

So what's the quickest and easiest way?

A migration:

class AddStoredProcedures < ActiveRecord::Migration
  def self.up
    # function distance(lat1,lng1,lat2,lng2)
    #   sqrt((lat2 - lat1)^2 + (lng2 - lng1)^2)
    # end
    execute <<EOS 
      CREATE FUNCTION Distance(lat1 FLOAT, lng1 FLOAT, lat2 FLOAT, lng2 FLOAT) 
      RETURNS FLOAT
      DETERMINISTIC
      BEGIN
        DECLARE dist FLOAT;
        SET dist = Sqrt(Pow((lat2 - lat1),2) + Pow((lng2 - lng1),2));
        RETURN dist;
      END
EOS
  end

  def self.down
    execute "DROP FUNCTION IF EXISTS `Distance`" 
  end
end

Down sides?

I'm told that using migrations for stored procedures has one major drawback - that if you want to change the procedure later then it's annoying to maintain - and can be the wrong version etc... In this case that's not a problem. Pythagorus isn't likely the change any time soon, so I'm unlikely to need to update it.

But there is one annoyance about putting it as a migration, and that's the fact that it doesn't show up as part of the schema... which means our Test environment can't find it. :P

This is a bit annoying, and I haven't found an elegant solution to the problem, but to get the test suite running, I have found a hack...

Re-run the migration:

class ActiveSupport::TestCase
  # ... lots of guff

  # Run stored procedures to add to the test db
  require File.expand_path(File.dirname(__FILE__) + "/../db/migrate/20100528190204_add_stored_procedures.rb")
  AddStoredProcedures.down # clear it out first, to be sure
  AddStoredProcedures.up
end

It's ugly, but it works. You may note I have to run the "down" then the "up" - this is because ActiveSupport ::TestCase actually gets loaded for each type of test-case you are running (eg once for Unit tests and again for Functional etc) and the "up" barfs if you try to create a procedure that has already been created in the db.

Otherwise, it runs and lets you get on with your work...

if anybody has a more elegant, but unheavy solution to the whole issue of Stored Procedures in test dbs... please do let me know.

ActiveRecord - find in random order

So we wanted to be able to find a few, random products to put on the homepage of one of our client's sites. I went looking for solutions, and the most commonly spouted one is to do a find with "order by 'rand()'"...

Now, this does the trick, for sure, but the problem is that it will order the entire table by the random amount... and we have a table with many thousand products, and this leads us towards a less-than-optimal query-time - every time somebody hits the homepage. :P

Given we only wanted a handful of products each time - I figured there just had to be a way of pulling out a small set of randomly-selected products without killing the load-time.

then I stumbled upon this article on Random records in rails. It provides a quick-trick fix that will pull out a single random record without all the fuss of ordering the entire table...

...but it doesn't provide a complete, extensible solution. We need it to pull out more than one - and I don't want to have to hit the database multiple times, if I can find a way around it.

Also - we don't just want *any* random product. Some of them are archived or out of stock, and so we need to be able to pass in other finder-option or use our nifty named scopes (eg "in_stock" or "best_sellers").

So here's my new solution. It lets you choose the number of random records to return, and pass in options, and plays nice with named scopes.

 
    # pull out a unique set of random active record objects without killing
    # the db by using "order by rand()"
    # Note: not true-random, but good enough for rough-and-ready use
    #
    # The first param specifies how many you want.
    # You can pass in find-options in the second param
    # examples:
    #  Product.random     => one random product
    #  Product.random(3)  => three random products in random order
    #
    # Note - this method works fine with scopes too! eg:
    #  Product.in_stock.random    => one random product that fits the "in_stock" scope
    #  Product.in_stock.random(3) => three random products that fit the "in_stock" scope
    #  Product.best_seller.in_stock.random => one random product that fits both scopes
    #
    def find_random(num = 1, opts = {})
      # skip out if we don't have any
      return nil if (max = self.count(opts)) == 0

      # don't request more than we have
      num = [max,num].min

      # build up a set of random offsets to go find
      find_ids = [] # this is here for scoping

      # get rid of the trivial cases
      if 1 == num # we only want one - pick one at random
        find_ids = [rand(max)]
      else
        # just randomise the set of possible ids
        find_ids = (0..max-1).to_a.sort_by { rand } 
        # then grab out the number that we need
        find_ids = find_ids.slice(0..num-1) if num != max
      end

      # we've got a random set of ids - now go pull out the records
      find_ids.map {|the_id| first(opts.merge(:offset => the_id)) }
    end