Software: no compensation necessary

I've just been reading "The World Is Flat" by Thomas L Friedman. It's an interesting book with lots good to say[1] about how the world is rapidly flattening - ie everywhere is becoming reachable from anywhere.

Along the way (and only incidentally to the main point) the author brings up the concept of Open Source Software.

He discusses the concept in a fairly positive light. He seems quite fascinated by the phenomenon - though doesn't seem to really get why people would do it. Though he makes a valiant effort to see things from the POV of the average OSS contributor, it's still fairly clear that he thinks it's a bit of a quaint, idealistic example of altruism (recognising the "pay it forward" aspect), rather than an effective and useful business technique. This is exemplified by the fact that he interviews a Microsoft Manager to ask whether or not it is sustainable as a business concept. Not bothering with an Open Source advocate, just an employee of one of the biggest proprietary systems out there.

One point stands out from what this employee says about OSS. He states that money is the main motivator for building software. From this the author concludes that any "really good software" will more likely be built by big software houses (eg Microsoft) than by OSS initiatives... after all, they have a lot of money, that can buy a lot of development time. He therefore concludes that The big software houses will always build the best software - simply because they can sustainably pay the most developers.

It's an interesting theory, and the most logical for those who don't really understand the motivation of the average OSS contributor. The reasoning behind this conclusion is of itself quite interesting. They claim that software is developed because businesses need stuff to be done - and thus they pay developers to do it. Nothing wrong with that... but they seem to mistake the effect for the cause and assume that the motivator for software to be built - is the money.

This is not the case.

Software is susceptible to the old phrase of "Necessity is the mother of invention" - ie the cause of software to be built is the business's need for it. The money is a secondary affector - ie the need causes the business to find money to make it happen. The originating need will be present regardless of whether the business has money to pay for developers... after all, even if the business has no money, the need will still be there for the software.

If the business happens to be made up of developers, then they are quite capable of creating the software, regardless of payment... and that is how you get startups: built and run by developers who see the need but don't necessarily have money to pay other developers - they just need the software to get their own business running, and are ready and willing to work in their own spare time to make it happen (nothing wrong with Sweat Equity).

The Open Source world calls this motivation "scratching your own itch" and it is recognised as the primary motivator for Open Source contribution[2]. Even for the really big projects.

Linux, for example, was Linus Torvalds' project to build himself an OS that didn't suck. Since he released it, thousands of other individuals have contributed to it for the same reason. It didn't do what they needed, so rather than bitching about it, they put their money where their mouth was and wrote the fix themselves... contributing it back to the OSS community... for no compensation except the "pay it forward" principle.

Which brings me to the other book I've just finished reading: "Drive" by Daniel Pink. One of the principle findings of this book is that fiscal compensation actually gets in the way of intrinsic motivation[3]. You can incentivise a developer to build something, but then they dissociate their ownership of the end product to the person doing the incentivising... care factor drops radically and you end up with... well... Microsoft products. Corporate and cold, buggy and with no real way to guarantee that the problems will ever get fixed.

This difference in motivation easily accounts for the fact that MS products are made primarily for creating more money for Microsoft (for which purpose they perform admirably), but only a passing interest in actually being fit for use: ie only so much as they can get away with before too many people stop purchasing them.

Now, there's nothing inherently wrong with that. More power to them for staying alive so long, and rather profitably - by keeping so many people satisfied for so long with what they're prepared to shell out... but there's a reason that I haven't used Microsoft products for over a decade now...

By comparison, the OSS world is driven purely by user requirements. If software is not fit for purpose, it simply won't be used... but more importantly, because there's nobody paying for it - if it doesn't do what it's supposed to, it will never be born

Conversely, if it is built, you can guarantee that it meets the requirements of at least *one* real customer (unlike many products built purely with potential revenue as a motivator).

OSS lives and dies on it's own merits... no compensation necessary.

Notes:

[1] Though I'd take a lot of it with a pinch of salt as well. This guy is clearly of the "yay capitalism, it solves all the world's ills" school of thought... though is slightly tempered by realising that humans are human, and some of them actually need supporting by those who are better-off... it's an interesting act of comparison to read this side-by-side with "The Shock Doctrine" by Naomi Klein - just as equally biased in the opposite direction, but also an fascinating read. I think if I ever put these two books together on a shelf, they'd annihilate one another with a very loud bang.

[2] If you're interested in this topic, "The Cathedral & the Bazaar" by Eric S. Raymond is an excellent book that explains the difference between what can be built from the ground up by need-motivated developers rather than the top-down software built by big software houses spending money to pay for what they want.

[3] which can be paraphrased as that innate curiosity and enjoyment of a challenge that occurs when you take on a new puzzle

Email campaign tracking with Silverpop

SilverPop offer themselves to the world as an "Engagement marketing solution". They provide click-stream and conversion tracking based on your email marketing campaigns, by putting unique identifiers in the links of your mailings (which they can send out in bulk). If you store these IDs and ping them back to SilverPop's click-stream and conversion-tracking servelets, this means you can get complete conversion analysis based on your email marketing campaigns.

I'm currently working for Moneyspyder, and one of our clients has signed up for a Silverpop account - which means we need a way to easily integrate with SilverPop - and the examples they give are all in php (and a little less than dynamic). So here's my Ruby-on-Rails solution.

1) Cookify the IDs

When a customer clicks on a link in one of the SilverPop emails - Silverpop has cleverly adding tracking IDs to identify the mailing-job and individual customer. When we send tracking information back to SilverPop - we need to send back these ids so the click-tracking is stored against the correct campaign and individual.

Of course, the parameters will disappear after the first time the user clicks another link, and we need to persist the data - to keep tracking all the pages they click on until they eventually convert!

The easiest way to keep track of this to stick it all into the session - then we can just check if it's there and send it each time we want to ping to SilverPop... and of course the best place to do session-wrangling is in a before_filter. So stick the following into something like application.rb

  before_filter :save_silverpop_data_in_session

  # This method does the storing of the ids from the URL into a session
  # cookie for later sending.
  # It will replace any existing values in the cookie - which represents the
  # user having returned to the site after viewing (and clicking on) another
  # link from a different email campaign.
  def save_silverpop_data_in_session
    silverpop_params = %w{spmailingid spuserid spjobid spreportid}
    if silverpop_params.all? {|f| params.keys.map{|k|k.to_s.downcase}.include?(f) }
      params_down = {} # ugly but necessary to get past browser case insensitivity issues
      params.each_pair {|k,v| params_down[:"#{k.to_s.downcase}"] = params[k]  if silverpop_params.include?(k.to_s.downcase) }
      session[:silverpop] = {:m => params_down[:spmailingid], :r => params_down[:spuserid],
                             :j => params_down[:spjobid], :rj => params_down[:spreportid]}
    end

  end

2) Configure your pod

SilverPop calls it's servers "pods" - and you could be using any one of them. This works better as a configuration option than hard-coded in your code - and lets you set up a link to the test-server on your dev/test environments. but in environment.rb you'll have something like this:

  # SilverPop URL = mailing list manager/ClickStream Analysis
  SILVERPOP_SITE_URL =  "recp.mkt41.net"
  SILVERPOP_SITE_HTTPS_URL =  "marketer4.silverpop.com"

3) Helping hands

Next up is to figure out how to ping SilverPop... which they helpfully give us an img tag example of how to build up the correct url with all the requisite values.

SilverPop has two servelets - one for accepting pings for click-stream tracking and one for the conversions. But they're almost identical, just taking different parameters... and I'm lazy and don't want to have to remember all the common details of how to do this in each place in the site. Thus: helpers to the rescue!

  # This method generates the code that calls the ClickStream tracking
  # servelet - we pass in the page name and page URL, along with the values
  # passed through from the last silverpop email - saved in the session
  def silverpop_click_stream_ping(page_name, page_url)
    silverpop_link('cst', :name => page_name, :s => page_url)
  end


  # This method generates the code that calls the Conversion Tracking
  # servelet - we pass in the "action", "detail" and "value" flags - 
  # eg "silverpop_conversion_ping 'CompletedOrder', order.id, order.grand_total" 
  # along with the values passed through from the last silverpop email -
  # saved in the session
  def silverpop_conversion_ping(action,detail,value = nil)
    silverpop_link('cot', :a => action, :d => detail, :amt => value)
  end


  # generate the SilverPop ping-image based on required servelet and parameters
  def silverpop_link(servelet, options)
    # skip out early if this user hasn't come through a silverpop email
    return nil unless session[:silverpop].present?

    base_url = (/https/ =~ request.protocol) ?  "https://#{SILVERPOP_SITE_HTTPS_URL}" : "http://#{SILVERPOP_SITE_URL}" 

    image_tag "#{base_url}/#{servelet}?#{session[:silverpop].merge(options).to_query}", 
      :height => 1, :width => 1, :alt => ""
    # we'd like this alt-tag: "Silverpop #{servelet.upcase} Servelet Ping"
    # to be accessible... but the above is not *really* an image, and thus will display 
    # the alt-text in the html at present... this sux, but I have yet to talk to silverpop 
    # about a way around this.
  end

4) Click-stream - analyse!

So, now it's time to get down and dirty with the click-stream analysis. This couldn't be more simple. Just use the helper to pop a link into your main layout. eg:

  <!-- silverpop cst ping -->>
  <%= silverpop_click_stream_ping(@page_title, url_for(:only_path => false)) -%>

Now, as you can see, this mainly works by using our dynamic page-title and a link to the current-page (using the empty url_for trick). Note that if you leave off the "only_path=false" option it won't provide the hostname. This may be what you want if you want to roll up multiple mirrored domains. You'll also have to adjust the page-name parameter as necessary for the way you generate the page-title... but otherwise you're good to go and this means every page-click gets tracked back to SilverPop from now on.

With one caveat... if you have AJAX-updating, you may need to figure out a neato trick of putting the ping-link into the newly-generated page-pieces or these "clicks" won't get tracked as the layout won't see them as new pages. I'll leave that as an exercise for the reader as it's very site-specific.

5) Mine your conversion gold

Now only the final, and most important, part is left - tracking actual conversions.

A lot has already been written about what constitutes an important conversion for your site. I won't repeat it all here as you can track heaps, and it's really down to what is important for your business. So I'll pretend we only care about when a customer completes an order - which we know because they land on the "thank you" page.

Which means we need to pop a link to the COT servelet there and pass in the important details... nothing easier:

  <!-- silverpop cot ping -->>
  <%= silverpop_conversion_ping("Order Complete", @order.id, @order.grand_total) -%>

Now SilverPop will monitor our marketing mails from go to whoa - and even know which order they completed and, most important, how much money they ended up giving us... Gold!

Update: Sorry for the repost (anyone that noticed). I fixed a few bugs after I finally got a chance to fully test this out.

Anybody implementing a Silverpop solution should also be aware that Silverpop's servers (at this moment) do not correctly redirect links from mailings sent from their test server. Instead you get redirected to your website, but without any values in the required query parameters (eg spMailingID etc are empty).

To do an end-to-end test of SilverPop's link redirection, you must create a real newsletter mailing on the live server, and just restrict the recipients (eg send it to yourself and any other devs on your team).

Stored Procedure-fu

So, I've been tuning my db performance on our heaviest database call... and this involved adding a quickie stored procedure to make life a little simpler.

It's nothing complex, just does a really simple distance calculation (using pythagorus)... which we needed because most versions of MySql don't come fully-loaded with all those nice functions that you can find in the manual.

Now the problem with stored procedures, is how to add them to the db without too much hassle. I have heard there are gems out there - but they seem a little like overkill, and the ones I've seen have been "build it on the fly"... I just want to put it into the db and leave it there.

So what's the quickest and easiest way?

A migration:

class AddStoredProcedures < ActiveRecord::Migration
  def self.up
    # function distance(lat1,lng1,lat2,lng2)
    #   sqrt((lat2 - lat1)^2 + (lng2 - lng1)^2)
    # end
    execute <<EOS 
      CREATE FUNCTION Distance(lat1 FLOAT, lng1 FLOAT, lat2 FLOAT, lng2 FLOAT) 
      RETURNS FLOAT
      DETERMINISTIC
      BEGIN
        DECLARE dist FLOAT;
        SET dist = Sqrt(Pow((lat2 - lat1),2) + Pow((lng2 - lng1),2));
        RETURN dist;
      END
EOS
  end

  def self.down
    execute "DROP FUNCTION IF EXISTS `Distance`" 
  end
end

Down sides?

I'm told that using migrations for stored procedures has one major drawback - that if you want to change the procedure later then it's annoying to maintain - and can be the wrong version etc... In this case that's not a problem. Pythagorus isn't likely the change any time soon, so I'm unlikely to need to update it.

But there is one annoyance about putting it as a migration, and that's the fact that it doesn't show up as part of the schema... which means our Test environment can't find it. :P

This is a bit annoying, and I haven't found an elegant solution to the problem, but to get the test suite running, I have found a hack...

Re-run the migration:

class ActiveSupport::TestCase
  # ... lots of guff

  # Run stored procedures to add to the test db
  require File.expand_path(File.dirname(__FILE__) + "/../db/migrate/20100528190204_add_stored_procedures.rb")
  AddStoredProcedures.down # clear it out first, to be sure
  AddStoredProcedures.up
end

It's ugly, but it works. You may note I have to run the "down" then the "up" - this is because ActiveSupport ::TestCase actually gets loaded for each type of test-case you are running (eg once for Unit tests and again for Functional etc) and the "up" barfs if you try to create a procedure that has already been created in the db.

Otherwise, it runs and lets you get on with your work...

if anybody has a more elegant, but unheavy solution to the whole issue of Stored Procedures in test dbs... please do let me know.

Blogger has a new template editor...

...I couldn't resist.

So my blog is now more ruby-ish. :)

PS: if you use the template editor... don't forget to add back stuff you customised in the template (eg your google analytics widget) *blush*

Alternatively make them into page objects so they aren't a part of the template at all

How *NOT* to do customer service

1. Start by telling your customer the work is not good enough. That the resolution will make it pixellated and that will look terrible.
2. Then say that the image has "keylines" that have to be removed and "crop lines" added "outside the trim area" - without explaining what they are
3. When your customer naiively asks what these mean, instead of explaining it - tell her that you haven't yet received the artwork because the only files they received don't have these things
4. When the customer says she sent files through and asks what is needed to change them into files that will meet their requirements (giving a detailed guess at what that might be), tell her again that you need these things - without explaining exactly what is needed or confirming that what she suggested is actually what you needed - even if she's later proved correct. Tell her that she is running out of time to get this done...
5. When your customer calls, make sure your receptionist (or colleague) answers, and that she tells your customer that you are with some other important client... and get *her* to also tell your customer that the artwork isn't good enough and go on at length about how it's such low resolution that it'll look all pixellated and low quality that it'll look terrible. Make sure she promises that you'll call her back before close of business.
6. When your customer has received no calls or emails and it's near close of business... and calls you. Answer immediately and...
   1. Begin by telling her off again at the low quality of the images (due to the resolution).
   2. When she mentions she thought you'd be calling her back, tell her you have *so* many other clients and you're very busy, you know...
   3. When she asks if you'll please just explain what she needs to do to add the crop lines - don't explain, tell her that you were expecting to speak with a designer who should already know these things.
   4. When she suggests that not all clients know the industry jargon of printing businesses and again asks for you to explain what to do, tell her to go and look up "crop lines" in the dictionary
   5. When she says that sometimes in your industry you have to deal with people that aren't specialists (she gives an example that if you'd asked her to set up a website, she wouldn't expect you to know what Rails, MySQL or AJAX were)... then tell her that you simply don't have time to explain to your customers how to add crop lines to an image, and that you only deal with designers that already know what crop lines are...

background...

We're launching our new startup this weekend (ie tomorrow) and needed new business cards ASAP. So my colleague called around the local while-u-wait business card dealers and found one that was willing to print and ship in time for tomorrow.

I'd quickly thrown together some business card images that were the exact size/shape required. They aren't brilliant, but they'll do for the weekend after which we can get some better ones done up.

I have a day-job... so while I'd love to go out and find out all about crop lines and trim-borders and whatever... I need to actually spend the day making money for my employer...

When I got off the final phone call and calmed down for a few minutes, I managed to find some time to go look it up. Apparently, all I need to do is exactly what I suggested in the reply to the second email - ie, expand the image size and add borders a little larger than where the old image-shape used to be... this would have taken him all of one minute to explain - just as it did me just now.

I recommend that you *don't* purchase from Kall Kwik 781 in Eton Street, Richmond

Shower ambassador!

A shower company called Grohe came up with a brilliant Social networking strategy that has people talking about their products.

They began by setting themselves up with a Grohe facebook fan page, with a competition. 1000 people would become "shower ambassadors" for the company - sent one of their stylish rainshower icon showerheads as long as they filled in the standard "tell us why you deserve to be a shower ambassador" forms.

I dutifully filled it in - I thought they looked kinda funky, and free stuff is always neat... and then forgot all about it, until yesterday morning when the postie arrived at the door with a shiny new shower head!

Awesome!

So, I've tested it and I think it's good. It's very stylish and would look right at home in most bathrooms - you can pick from a wide range of colours to suit. It's definitely a "rainshower" style head: very gentle; and eco-friendly as it uses less water for a very broad effect. It even has an extra-eco-friendly button that lets you use even less water... and the head comes with this set 'on' straight out of the factory - so they seem like they're serious about being eco-friendly.

Of course the really clever thing about this was the campaign itself. From the information they've posted me, it's clear they intend to get the message spread far and wide. The shower head came with a page of information suggesting that you share "thoughts, pictures and videos on our facebook page... your blog or your Twitter account".

Through the competition, they've made it a fun thing to do and share, and I wouldn't be surprised if they picked people solely on their social-media-integration. Apart from running their competition through facebook I vaguely recall they also had fields for your other social media accounts... which I'm sure is why I got picked.

Clearly these guys are switched on the the power of viral marketing - and aren't afraid to ask people to do it... but it's more than that. They also seem to realise that the best way to get people to spread the message is to put their actual product in the hands of the people most likely to spread the idea.

They've also got the be remarkable idea down pat. They clearly picked a product that was wroth talking about: an iconic, shower-head with a vibrant colour and notable green-credentials. Part of their success here, was making it *personal* to the individual participants. When filling out the competition-form, you got to pick your favourite colour[1] showerhead. This is such a small thing to do - but with great effect as it lets people feel they have the power to choose. I don't think most companies would think of doing this when it comes to items that they are, remember *giving away for free*. But it's such a powerful and brilliant strategy, when you think about it.

Grohe want people to talk about their product - and especially they want them to talk about how much they *love* this product... Letting a person choose their preference (just like they were buying it), means that it isn't just some random item that rocks up at your door, but something that *you chose* and is already personal before it even appears - thus increasing the likelihood of giving it a thumbs up...

All in all, I'm quite impressed by their strategy. Moreso even than the shower-head itself - which is, btw, quite funky.

[1] I picked their dark-blue - I like blue bathrooms.

Truth vs Harmony

People live on a spectrum of preference. Some prefer to hear the hard truth, regardless of whether it will hurt their feelings. Some prefer a comforting lie, feeling that harmony is more important than being right.

Both views have their pros and cons, and different situations generally call for a different balance on this particular scale. Still, people tend to have a preference for one or the other.

People often prefer different levels in different circumstances eg when we're more emotional, we tend towards comforting lie, and people will generally react to that - keeping any harsh truths for when a person is more able to bear it. Lying to children is often rationalised in this way - falsely in my opinion.

People at opposite ends of the spectrum often simply can't understand what the other people want. What motivates them to prefer a way of dealing with truth so different from their own?

Harmony people can't understand why a person would tell the truth if it might be more hurtful. They'll say, with perplexity "what they don't know won't hurt them, but the truth will hurt their feelings, so why do it?"

Truth people feel like they have no control if people keep lying to them, they'll ask: "They deserve to know the truth, after all, how could I make a decent decision if I can't trust what people say to me?"

Harmony people believe it doesn't matter as long as everyone gets along. Life is about getting-along, after all. Often-times, telling the truth causes them to lose face, which doesn't help anybody. But are they just trying to avoid fixing a problem that's their responsibility to fix? Are they avoiding a truth within themselves because they're too lazy or ashamed to face it?

Truth people feel robbed of a valuable chance to learn something if not told the truth. They feel that they are doing a service to another person by giving them a chance to see themselves and learn something. But are they just nitpicking, poking at wounds or being too lazy to be tactful?

In a bizarre twist, many people will say they prefer truth, but are actually lying. It's more comforting to believe you are a hard-hitting, strong person - able to handle any truth... it's much harder to face up to the truth that you are in fact not able to face up to the truth.

Personally, I've always preferred the truth. Yes, it can be hurtful. Yes - its embarrassing and sometimes degrading to know the negative effects of you impact on the lives of others, and there are definitely times and places where it's better to bite your tongue; but unless pressed, I'll always opt for the truth.

Both giving and receiving.

This can be hard for harmony-people to understand. Just as it's hard to explain that introversion[1] is not actually unhealthy, just a different way of being.

I'll usually get a response along the lines of: "but life is about enjoying yourself, not being right". Saying that I cannot enjoy myself if I know there's something I'm doing wrong, or even "It's just the way I am" generally doesn't help much. Harmony-people still often shake their heads and seem to think there's something unhealthy about the whole thing.

I've had more success when I explained that for me, improving myself is something I like to do. I work hard at it, and try to strip myself of self-delusions (though I'm under no delusion that I've got rid of them all). I am not on the quest for hedonism, but the quest for self-improvement...

So how does that fit with truth? Well - if I do something wrong, and you tell me it's ok - just to keep the peace... you're depriving me of my right to learn from my mistakes. That makes me feel upset.

I can appreciate that others prefer the opposite. They are trying to enjoy life and get through with a minimum of fuss and anguish. Most people seem to feel that if there's something nasty, it's better to coat it in shiny mother-of-pearl so that it's not all sharp and spiky in your side.

I can't help feeling, though, that it'd be much easier in the long run if they just faced up to it and spat it out to begin with. Then whatever it is would be gone from their lives and they could move on...

It often seems to me that people make their lives much more difficult by not facing up to hard truths, but by trying to ignore them or put them off. To convince themselves of the comforting lie in a misguided effort to "make it all go away". We all recognise these in other people... "I'm just big-boned", "I'll get around to it someday", "maybe it'll get better"... but when it comes to ourselves it's much harder, because we realise it's *us* that has to do the work to get past whatever it is... instead of somebody else's time/effort (which is as easy to spend as other people's money).

I am not immune to this myself, but whenever I find it happening in myself I tend to make an effort to getting rid of whatever it is I'm deluding myself about.

Obviously both can be abused. Lying to somebody can lead them into trouble, and telling too harsh a truth can indeed be needlessly hurtful - especially if a person is vulnerable, and simply needs comfort. Telling the truth about a person in front of other people can degrade their worth in the eyes of the others (and themselves). This is a Bad Thing.

Still, covering for another person's bad behaviour can mean that a person is not stopped from doing something they shouldn't - or never realises that they're being inappropriate (no, they don't always know!).

There is a place for truth and a place for tact, and it's a hard line to walk between the two.

IMO, though, society currently seems to place a heavy burden on the "play nice" aspect, and not enough on "keeping each other accountable".

The other argument about this is the lash-out defense of "it's none of your business". Who am I to tell other people to shape up? Especially since I'm so flawed myself. is this a "log in thine own eyes" situation?

To a certain extent this is the case. I certainly shouldn't feel high and mighty simply because I see a speck in your eye... but neither should I let you needlessly destroy your vision if you aren't aware of the issue.

It's absolutely my business to tell somebody that they need to learn to pick up the phone if leave me waiting for hours in the middle of nowhere, after they said they were going to pick me up. That situation doesn't have much of a grey area - but there are a lot of areas where it's more cloudy.

It's none of my business that a person is deluding themselves that their latest fad diet will work just as badly as all the others... unless they happen to be a friend that I care about and don't want to see them being unhealthy and eventually unhappy when their diet fails to work... again.

Is it my business? Yes and no. Should I find a tactful way of telling the truth? sure thing... but it will be the truth. The comforting lie will actually by more hurtful in the long run, to her health, to her self-esteem, to her chances of success at the very goal she has chosen for herself. In this sense, the comforting lie seems to only be less-hurtful in the short-term.

I believe this is true of most comforting lies. They cover-up a hurt to make life easier in the short-term, but the long-term harm is far more pronounced. On balance, I believe that only the truth will set you free.

Notes

[1] Introversion in the Myers-Briggs sense, an introvert is a person who regroups and regains their strength when they are by themself, rather than an extrovert who does so when in company. It is not actually the same thing as "being withdrawn" or even "being socially backward" - though these can be a consequence of too little interaction with others.

[2] Lying to children I think children are just as capable of bearing truths as adults, so long as they aren't frightening. Things that may make an adult fearful need not be so to children. eg death is a fact of life, but it needn't be something feared - of course you don't explain the scary details to a child as that will make it fearful.

ActiveRecord - find in random order

So we wanted to be able to find a few, random products to put on the homepage of one of our client's sites. I went looking for solutions, and the most commonly spouted one is to do a find with "order by 'rand()'"...

Now, this does the trick, for sure, but the problem is that it will order the entire table by the random amount... and we have a table with many thousand products, and this leads us towards a less-than-optimal query-time - every time somebody hits the homepage. :P

Given we only wanted a handful of products each time - I figured there just had to be a way of pulling out a small set of randomly-selected products without killing the load-time.

then I stumbled upon this article on Random records in rails. It provides a quick-trick fix that will pull out a single random record without all the fuss of ordering the entire table...

...but it doesn't provide a complete, extensible solution. We need it to pull out more than one - and I don't want to have to hit the database multiple times, if I can find a way around it.

Also - we don't just want *any* random product. Some of them are archived or out of stock, and so we need to be able to pass in other finder-option or use our nifty named scopes (eg "in_stock" or "best_sellers").

So here's my new solution. It lets you choose the number of random records to return, and pass in options, and plays nice with named scopes.

 
    # pull out a unique set of random active record objects without killing
    # the db by using "order by rand()"
    # Note: not true-random, but good enough for rough-and-ready use
    #
    # The first param specifies how many you want.
    # You can pass in find-options in the second param
    # examples:
    #  Product.random     => one random product
    #  Product.random(3)  => three random products in random order
    #
    # Note - this method works fine with scopes too! eg:
    #  Product.in_stock.random    => one random product that fits the "in_stock" scope
    #  Product.in_stock.random(3) => three random products that fit the "in_stock" scope
    #  Product.best_seller.in_stock.random => one random product that fits both scopes
    #
    def find_random(num = 1, opts = {})
      # skip out if we don't have any
      return nil if (max = self.count(opts)) == 0

      # don't request more than we have
      num = [max,num].min

      # build up a set of random offsets to go find
      find_ids = [] # this is here for scoping

      # get rid of the trivial cases
      if 1 == num # we only want one - pick one at random
        find_ids = [rand(max)]
      else
        # just randomise the set of possible ids
        find_ids = (0..max-1).to_a.sort_by { rand } 
        # then grab out the number that we need
        find_ids = find_ids.slice(0..num-1) if num != max
      end

      # we've got a random set of ids - now go pull out the records
      find_ids.map {|the_id| first(opts.merge(:offset => the_id)) }
    end

current_user in rubyCAS-client

With restful_authentication (or other schemes) we have a requires_login before_filter which you stick at the top of your controllers, to make sure certain actions are protected by login. It also tends to take care of the slightly messy logic involved in setting up the extremely useful current_user variable.

Obviously our applications have already been built around this logic existing, so a smooth transition requires the same sort of functionality.

The ruby-CAS (client) provides a basic before_filter for us that takes care of the nasty talking-to-the-rubyCAS-server logic... at the end of which we know whether or not the person talking to us is known to the system. But we still to actually associate this person with a real live User model on our system, and save them into the current-user variable.

On our system, after going through the provided before_filter, we have configured the server to provide two things in the session:

1. the :cas_user, which is the user's username and
2. a set of :extra_attributes which is a set of other fields we've asked CAS to provide and which we'll use later for roles (see authorisation for more on that)

If we want any other user information - we will need to go fetch it by hand from our local db, and we'll need to do this directly after the main rubyCAS before_filter. So, we'll need something like the following, probably put into ApplicationController

  # this is the before_filter provided by rubyCAS client
  before_filter CASClient::Frameworks::Rails::Filter
  # this is a call to our own new before_filter
  before_filter :setup_cas_user

  # actions go here...

  # and at the bottom of the controller file...
  private
    # this before_filter takes the results of the rubyCAS-client filter and sets up the current_user
    def setup_cas_user
      # save the login_url into an @var so that we can later use it in views (eg a login form)
      @login_url = CASClient::Frameworks::Rails::Filter.login_url(self)

      # if we don't have the :cas_user in the session, then we are not logged into rubyCAS
      # we could have none because:
      #   1) we have failed login or 
      #   2) because we don't necessarily need to login to get here
      # either way we need to skip out here
      return unless session[:cas_user].present?

      # so now we go find the user in our db
      @current_user = User.find_by_username(session[:cas_user])
      @current_user.present?
    end

Pretty simple

What if the user isn't on this application yet?

Good question - it is possible to have created a user on one application and of course, using single sign-on, you may want to automatically sign your users up into another application. This could easily be done here with a slight addition to the before_filter:

      # so now we go find the user in our db - or create them if they don't yet exist.
      @current_user = User.find_or_create_by_username(session[:cas_user])
      @current_user.present?

Note: the only information linking the two user-records will be the username. You won't be able to access any other information (eg address or other details) unless you physically query your other application's user record... though this is a good use-case for passing extra information through the extra_attributes fields. Still, it does allow your users to seamlessly transition from one of your applications to another.

This is one article in a series on Rails single-sign-on with rubyCAS

RedBubble

I have just discovered RedBubble an awesome site for displaying and selling your artwork (whether images, writing, T-shirts or otherwise)... and developed in Ruby-on-Rails too.

You can upload your art and offer it for sale in a variety of formats - they provide widgets for helping sell your stuff (eg the portfolio below). You just specify the markup you want and they take care of the shipping and payment - sending you your cut via checque or paypal. It also has groups a-la flickr - where you can join your art with other similar art, take part in challenges etc.

Here's my portfolio as an example of what they can do for you.

Silicon Stilettos

I just went to a great meetup. Silicon Stilettos is a networking meetup aimed at business women in the IT industry.

I went along last night and loved it. It was a great way to catch up with like-minded women in the industry, and hopefully learn a few tips and tricks from them, and share my own experience.

Despite it being an IT industry Do, I seemed to be the only actual developer in the lot - but that didn't surprise me at all. Even after decades of equal-rights, I still find that even with the small number of women in the IT world, that majority of them tend to be in marketing or PR, which played out amongst this group. But that's fine - after all, if I wanted to learn yet more IT stuff, I could go for linuxchix, GirlGeekDinners or any one of the non-female-only[1] user groups I attend already. So it was great to see a different side of the IT industry - and there were certainly a few women there that were quite savvy in the field.

The best chat was about social media as it pertains to IT PR, how big corporates still haven't realised they should do it, and how those that realise they should don't know how to get started. Mainly because they're so used to hiding behind their corporate brand-projection that it's hard for them to dredge up a real, unique and human voice, which is what is needed for social media to actually be genuine and appealing.

Often, big companies try to continue the sort of push-style marketing they're used to, just dropping it into twitter/facebook as though that was how to win over the social media crowd. They couldn't be more wrong, of course. Social media demands a real and genuine connection with people, and you can't do that by braying about how good you are.

A lot of companies still don't get the "conversation" part of the equation and fail to understand that at least one half of a conversation should be devoted to really listening to the other person[2]. Failing to pay attention can lead to serious problems quicker than you think. One of the women spoke about some kerfuffle that happened over facebook/twitter when someone at nestle was obviously having a bad day. Clearly, social media can also go horribly horribly wrong...

So, I think the meetup wa a resounding success, and I'm really looking forward to the next one.

Notes

1
Why go to an all-women group? Well - that's hard to answer without offending people, but I'll try. It's not that I don't like co-ed groups - I'm not terrified of men. Being in an industry so obviously dominated by them means I've gotten pretty used to hanging around with them... but there are still cultural differences between the sexes (in an on-average way, and I'm sure entirely due to different upbringing). While it's great to do the mixed-thing it can also be good to get amongst other women too.

Being an expat I can immediately see similarities with hanging out with others from my own country. It's not that I don't want to assimilate - I spend > 95% of my time hanging around with locals... but it's also nice now and then to get together with other Aussies and bitch about how crap the NHS is compared with Medicare (for example). :)

The expectations and culture are simply different. For one thing, I noticed that most of the attendees had one small drink and I wasn't the only one leaving before 9 to get home. Being used to IT get-togethers where you get funny looks if you don't have a second or third drink and get a sense that you're just "not up to it" if you don't hang around until way past 10 (regardless of it being a school-night)... this came as a breath of fresh air. Seems like these other women also have a life... :)

2
It still surprises me that companies don't get this. After all, The Cluetrain Manifesto was written a decade ago, and folks like Seth Godin are doing their best to get the message out every which way they can. Yet I still see big companies pushing the same rhetoric out as though they could just transcribe their traditional TV-ads or print-media brochure-ware onto the web. They remind me of those obnoxious types that will talk over the top of everyone at dinner but only ever talk about how good they are, without realising that the adage of "show don't tell" isn't limited to novel-writing.

What am I up to?

Wow, time goes by so fast!

It seems that only a few weeks ago I was writing my RubyCAS blogposts and now it's suddenly April and I haven't finished them up yet. They're all sitting, half-written in my drafts folder.

The past few months have been crazy-busy. I left my previous employment in Windsor and am now working two part-time contracts - one for Mobibliast in Australia one day a week (paying off the mortgage back home), and one for Moneyspyder three days a week in London. For my remaining one day a week, I'm working on my own startup - which I'll tell more about as it creeps closer to launch-date... though that one day a week often creeps over in to Saturday and sometimes Sunday too. :P

RubyCAS Logout action

Here's a quick snippet for the CAS logout action. It logs the user out of the CAS-server and redirects them to the root_url of your application.

  def cas_logout
    @cas_current_user = @extra_attributes = nil
    reset_session
    CASClient::Frameworks::Rails::Filter.logout(self, root_url) and return
  end

This is one article in a series on Rails single-sign-on with rubyCAS

Login to rubyCAS from a form

Now, as with most places, the root url for our site points at the signup-form. It's there to tempt new users to join us and start creating a new dating site immediately.
But our existing users will usually hit this page first too. Again like most other sites, we don't want to force them to click 'login' and go to some other page to fill in their credentials; so we have a login box on the signup form so they are one click away from getting to the real site.

Now, when you look at the rubyCAS implementation - all you're given is a login_url method that you can put into a link and sends you to the rubyCAS login page.

At first glance, this looks a bit like you're forced to do the login waltz:
click login... which takes you to the rubyCAS-server... you type your credentials into the box... you get redirected back to the site you really want... and one-two-three, and one-two-three...

Luckily there's a way to get your site to send credentials to the CAS-server from a controller-action, and redirect them straight on to the page they really want - so you can look like you're seamlessly logging them in without even touching the rubyCAS-server... and no, you don't even need to set your app up as a proxy server[1]. :)

The following shows how to use the existing configured CAS-server details to create a login action that will:

1. Send the given login details to the configured rubyCAS-server for the current environment
2. If it fails, will re-show the 'new' page (assuming that the login-form is on the same page) with a 'login failed' notice
3. If it worked, will redirect to a url that you provide... with the new CAS-ticket appended, thus properly logging the user in.

  # assuming that the signup-actions are in the same controller, 
  # you'll probably have a filter like this
  before_filter CASClient::Frameworks::Rails::Filter, :except => [:new, :create, :cas_login]
  def cas_login
    credentials = { :username => params[:login], :password => params[:password]}

    # this will let you reuse existing config
    client = CASClient::Frameworks::Rails::Filter

    # pass in a URL to return-to on success
    @resp = client.login_to_service(self, credentials, dashboard_url)
    if @resp.is_failure?
      # if login failed, redisplay the page that has your login-form on it
      flash.now[:error] = "That username or password was not recognised. Please try again."
      @user = User.new
      render :action => 'new'
    else
      # login_to_service has appended the new ticket onto the given URL 
      # so we redirect the user there to complete the login procedure
      return redirect_to(@resp.service_redirect_url)
    end
  end

At present, the two patches required to make this work are only available on my fork of rubycas-client, but I'm told they'll be pulled into master pretty soon.

Note on URLs

You have to pass in a url that the user can get to... and that is also protected by the CAS Filter. The login_to_service method really just generates a one-time ticket that can then be used on any URL that will accept that ticket (ie anything protected by the Rails::Filter) which is what does the 'real' logging-into the system. So make sure you redirect the user there... but only if it was successful (or they'll just end up at the rubyCAS-server page - which might not be a bad fall-back).

When considering what URL to start your user off with - remember that at the time we have to construct the url, we not only don't have a 'current_user' set up yet... we also don't know if the person asking for user Bob's page is actually user Bob. So if we want to redirect to the user's account page after login (eg users_path(@some_user) we have to a) guess the user's id from login, but also b) check that they could access the page *before* they successfully log in.

If you don't do this, you can potentially compromise security. For example, a malicious user might try a whole bunch of user logins. If the url is set for, say users_path(User.find_by_login(params[:login])) - that line will explode (with a routing error for trying :id = nil) for users that don't exist - conveniently telling our malicious user which logins exist (or not) on our system. This is generally considered a Bad Thing To Do.

A better idea is to have a generic url that doesn't rely on a specific id in the route. The action can pick up the current_user's id once the user is successfully logged-in. We use something like 'my_dashboard_url' or 'my_account_url' for this purpose.

---

Notes

[1] Setting up an app as a rubyCAS proxy server
This is something still in the eventual plans, but not yet. Mainly because it's a bit annoying that you have to have two apps running on separate servers simply so that one could be your proxy... that cross-dependency makes it a bit awkward to me. YMMV - the instructions are all in the rubycas-client rdoc

---

This is one article in a series on Rails single-sign-on with rubyCAS

Rails single-sign-on with rubyCAS

As our rails apps multiply, we began looking for a single-sign-on solution - to give some sense of a seamless 'application' to our users. We've picked rubyCAS for this as it's pretty much the only single sign-on solution that's already out there and running for ruby (and comes with both a server and client gem already written, which is handy).

We'd been running restful_authentication on our main app, and so our new CAS setup had to be able to handle all the existing stories before it was acceptable as an alternative. Over the next few articles, I'll share some of the more useful little snippets I came up with.

In a nutshell?

CAS is a protocol for Authentication. It is not exclusive to ruby, but the ruby-implementation consists of two halves: the server and client.

The server is an independent 'camping' application (ie rack-style ruby). The client is a gem/plugin that fits inside each of the applications that we are writing. The client has the code that allows us to use the rubyCAS-server as an authentication method. Most of the rest of the articles will deal with using the client.

How the two talk to each other is pretty complex, and most of it is beyond the scope of this article... and mostly you don't need to know the details to get it all working. Again, in a nutshell:

Users that try to hit one of our applications will be passed over to the rubycas-server for authentication which has a login form. They type in their login details. If this matches what's in our system, then they will be given a one-time ticket and sent back to the application they started at. The application will verify this ticket with the server, and get a proper ticket and put it into a cookie that will then allow them to be continually authenticated with that application from then on.

The single-sign-on part comes from when the user tries to then access one of the other applications running rubycas-client. The new app will see the user has the cookie, and send it back to the rubycas-server.. but gets redirected back to the new app before they get actually shown the login page with a new "it's ok, we know this guy" ticket. The new app sees this ticket and now user will be automatically logged into the other application as well.

If the user clicks logout, the app destroys it's own session, and sends the user to the rubycas-server logout page... which also destroys the ticket in its own database.

That's how it all hangs together.

If you really have a burning need to know more details, go check out the rubyCAS-server documentation which has full workflows.

So what do we want?

As I said, we're re-implementing the authentication that we had as restful_authentication... so we need to implement all the features we had there... but also single-sign-on to our new applications. What follows is a minimal wishlist for replacing existing functionality:

• Filter and Login action that populates local user (eg current_user)
• a logged_in? function usable for greetings and displaying the right login/logout buttons
• Logout action
• filter for pages that sometimes require login (ie the GatewayFilter)
• Using a form to post login-credentials to CAS (so you can put a login-box on your signup page)
• admin_login_required (access restriction before_filter making actions admin-only)
• Auto-login after create (so your newly-registered user doesn't have to retype the login credentials they just gave you)
• Redirection of root_url to a special page (eg dashboard) if the user is already logged-in without breaking all of the above
• Authorisation (eg user roles)
• Single sign-out (the issues and why we didn't...)

I probably won't tackle them in order... but should get to all of them eventually.

[Link] Bootstrapping passenger and Ruby Enterprise on Ubuntu

Michael Lang has provided a whole bunch of extremely useful scripts to help you install passenger and ruby enterprise on ubuntu. He's blogged about why he's provided the scripts - basically because he want everybody to bootstrap for his blogposts so they can start with the same environment set up before he goes into the details of a particular explanation.

I for one am simply grateful - but his comments are closed on the articles, so I guess I'll just have to point at him instead. :)

Thanks Michael!

Pretty error messages using locales

Default error messages are a set of concatenated strings all lumped together. This is great as a bare-bones set of errors, but does lead to clunky, unhelpful phrases such as 'Email is invalid'.

While technically correct, this can be a bit confronting, and doesn't explain to a user what they can do to fix it. After all, we already know that it's in our best interests to be nice to our users and get them up to speed with a minimum of fuss. Our error messages would be much nicer if we could use plain English and actually explain what a user can do to fix the situation. eg 'Please enter a value for Email that is longer than 5 characters. It should have an @ in it and make sure that you haven't accidentally used a comma instead of a dot'. That's not only nice and friendly - but means a user knows exactly what to do to check if they've got it right, instead of just getting frustrated.

So, how do we go about being nice to our users?

Adam Hooper has written a great article on using full sentences in validation messages. His reasoning is based on Il8n requirements which aren't necessarily everybody's cup of tea... but it's not just useful for that. It also gives us an easy way to make our error messages pretty.

The quickie version (for people that don't care about Il8n) is outlined below.

Create the locale file

All your error messages will go into the file: config/locales/en.yml.

Type in your alternative error messages in exactly the format you'd like them to appear to the user. There are special patterns that you can use to match the attributes of the error - the two main ones being the attribute name (which is substituted wherever you put {{attribute}}, or the required length of a field eg when used in validates_length of (which is substituted for {{count}}) . Here's an example of some of the more common Active Record errors rebuilt in this way:

en:
  activerecord:
    errors:
      messages:
        # Default messages: so we have complete sentences
        confirmation: "{{attribute}} doesn't match the confirmation. Please retype them and make sure they match."
        accepted: "Please read and accept {{attribute}} before continuing"
        empty: "Please enter a value for {{attribute}}"
        blank: "Please enter a value for {{attribute}}"
        too_short: "{{attribute}} is too short (the minimum is {{count}} characters). Please enter a longer one."
        taken: "Your chosen {{attribute}} has already been taken. Please choose another"

Remove any partial-errors in your models

You'll probably find that you've unwittingly entered partial error-strings in your code. eg errors.add :attr_name, 'is bad because blah blah blah'. So your next step is to search for these and turn them into real sentences too, or your error messages will be a bit out of synch. eg:

# instead of
errors.add :region, "should be from your chosen country. Please check it an try again"
# use:
errors.add :region, "The chosen region should be from your chosen country. Please check it an try again"

update the error-box to use only the error message - and not the fieldname

The last step to get your application nice is to stop using the auto-generated error messages, which automatically jam the fieldname up against the now-full-sentence errors. (producing such lovelies as Region The chosen region should be from your chosen country. Please check it an try again :P

This just consists of writing a helper method (or partial) that will take any kind of object and spit out the errors on it however you like. eg

  <div class="errors">
    <% @obj.errors.each do |f,msg| -%> 
      <li> <%= msg %> </li>
    <% end -%>
  </div>

And technically you're done...

except for shoulda...

As of my writing, Shoulda was not playing nice with locale-based error messages. My fork on git has the fix and I'm sure it'll be pulled into shoulda sometimes soon (let me know if it has!).

To use my branch, you an either vendor a copy of my branch, or add it as a :git => line in your Gemfile.

If you don't use the fix, shoulda will break because should_validate expects all of the old error messages :(

Getting webistrano to deploy under passenger

What's your problem?

I'd been playing with webistrano on my own machine and using it to make deployments to staging - and that was all ticking along nicely. But then it was time to put webistrano up on our staging server - so as to let our sysadmins use it for deployments.

Downloading and installing it is very easy. Configuring it is a little more annoying, but once it was full of all the hosts/systems/rstages/roles and config_options from my existing setup iit shouldn't need any more updating.

Then I tried to deploy. At which point it promptly all fell over.

The same thing happened over and over. I'd click deploy and it'd start. The "running" spinner would spin... forever, the little ajax refresh constantly refreshing the Log... with nothing.
The deploy didn't actually deploy, and what's worse - didn't ever *stop*[1].

I did a deploy and watched top and a ruby process did appear briefly and then disappear - but no ruby process ever showed up in ps...
Not happy Jan :(

I couldn't even cancel the deploy because the cancel button only ever appears once the deployment is fully underway[2]. To deploy again, I had to break locks and do other unsavoury things and that felt really wrong.

So, what was happening and how did we eventually fix it?

Well, the suspicious thing to me was that nothing showed up in the log-section at all. Not even the "loading stage recipe X" that is the very first line of all the successful deploys on my own system.

Thus I figured on a permissions problem. I checked that the runner-user had access to svn, and to the staging server. This was interesting as staging was deploying on itself, we checked that it could ssh into itself happily... and sure enough it asked me to check the 'authenticity of the host' I was connecting to. Now webistrano is notorious for hanging on an un-expected question, so I figured I'd just have to add this to known_hosts and all would be well.

It probably was something that helped... but the deploys were still failing to spin up.

So I dug into the log and found it was chock full of the AJAX Deployment#show refreshes (they're so frequent!) But I eventually got back to the initial Deployment#create which is what should kick off the real deployment. The log for this shows that it goes along fine until right at the bottom, almost completely hidden amongst the noise is one line that rang alarm bells:
sh: ruby: command not found

So I checked permissions again, checked to be sure that ruby was installed, that we could see it in the $PATH as the deployment user, all those things.
I even did a capfile export and ran it with pure capistrano to make sure - and that worked just fine! So now I was really confused.

Finally digging into the webistrano application code, I discovered that the important LOC is in app/models/deployment.rb under def deploy_in_background. It's of the form: system("sh -c \"cd #{RAILS_ROOT} && ruby script/runner -e... etc etc. So I tried this on the command line. ruby -v worked for the deployment user.

I spun up script/console and tried system("sh -c \"ruby -v\"")
and that spat out the correct version and 'true'... so obviously rails could find ruby ok, but running in during deployment was still not happy

Then I copied the above code into the application layout... and it returned false instead of true. Something was happening when inside the running app that wasn't running from the command-line.

Then I discovered this blogpost claiming they also had the log message: sh: ruby command not found

So it seems that when the app is running - by default you're actually not inside a shell - so it's not loading your settings (such as $PATH) and thus not going to find important things (like ruby).

The solution?

Instead of sh -c we need it run under bash --login -c

This will force the process to load up your bash environment settings. The bad news is that you have to monkey-patch webistrano to get it working[3].

Given webistrano is a rails app, this isn't difficult - just annoying. There's only one spot that you need to change. That's the aforementioned deploy_in_background method. Change it there and touch tmp/restart.txt and your deployments should at least spin up now.

anything more?

There is still problem if your recipes also require some $PATH goodness. For example if you are running 'gem bundle' your shell will need to find gem... which means that the recipes need to run under bash too. Now it's a little easier to convince webistrano to do that.

You can override the shell used here by supplying a configuration option: default_shell to bash --login

Note: it's the --login that gets it to do what you want!

Also - don't forget that if you call sh -c explicitly in your recipes you'll need to change it there too.

Notes for webistrano devs:

[1]You should probably surround the deploy process in a timeout.
[2] The cancel button should appear immediately.
[3] It'd be nice if we could configure the shell-command under which webistrano runs

The #1 thing to learn if you're new to git

Now I'm not 'new new' to git anymore, I've been using it off and on for at least a couple of years, but I've been using cvs/svn for many more years than that. git has a steep learning curve[1], and have found the git-svn crash course invaluable. But there was one thing missing:

The #1 thing I wish I had learned when I was new to git:

...is that you must commit to git *far* more frequently than you do to svn.

From my thrashing about, I have discovered there are many more 'dangerous' commands in git than in svn, and it's really easy to get yourself into a 'stuck'[2] state.

git actually provides a whole set of tools that will help you get back out of whatever hole you've dug yourself into... but you're likely to end up having lost your working-copy changes[3] along the way. So the best practice now is to commit often - so that everything is in the repository: there is never a big working copy to lose.

I had a lot of trouble with that as it considerably changes the workflow I'd built up over the last decade or so. From cvs to svn and then adding agile on top, my workflow is now roughly:

1. Checkout a fresh, clean copy of the repository (or svn up to achieve same effect)
2. add your tests and make some changes
3. check the rest of the tests still run - make changes until they do
4. do an svn st to see if there are any files I've forgotten to svn add - add them
5. do an svn diff and see all my changes - eyeball them to make sure there's nothing obviously wrong
6. do an svn up to pull in latest changes by others and make sure the tests *still* pass
7. svn commit

So at all times, everything would be *not* checked in - all of it just sitting in the local working copy until I was sure it was ready.

The problem is that, on the surface, git appears to support this workflow perfectly. All the svn-commands described above have git-equivalents (they're even called the same thing) and so you can (supposedly) transition smoothly over to git with only minimal effort. Even adding a branch, rather than hacking on master is not too far a departure from svn-style workflow, as branching is familiar in svn, and git just gives you a beezier easier interface.

So where does it break down?

Well, in my case, usually at the second-last point. A git-pull can completely mess you up if you get to a merge-conflicted state. You can't commit your working-copy because of the merged state, you often can't even properly diff because you've got a mush of the git pull's changes plus your own changes and no way to tell which is which. and there seems to be no obvious way to 'just commit the merge-conflict changes' or update the files that are conflicted and just *tell* git that they're not conflicted anymore... the way you can in svn. So at this point you're screwed.

What makes it worse is that at this point you often don't know exactly what commands you did to get you here - if you're anything like me, you've probably tried a whole bunch of stuff only partly understanding exactly what it does. Each command simply tells you in it's own way that you can't do that. You can look up what you're supposed to do to fix it - but generally find that's just another command that tells you that you can't do it either. So you feel like a truck that's stuck sideways in a narrow alley and can't even understand how it got here, let alone how to get itself back out.

Frustrating!

Underlying that is the, quite reasonable, fear that you may lose all your work[3] since your last commit...

and of course that's because we're used to the underlying 'don't commit until' mentality that we may not even be aware we are sporting.

don't commit until (perfect)

The workflow I described above is a perfect example of this mentality. It makes sense to hold back on committing anything until it all works. After all, you know that the moment you commit, the CI server will pull all your changes and let everybody on the team know that you just broke the build (again). So eventually you adopt a "don't commit until the tests pass" workflow, and keep everything in your working copy until everything's green before committing to the svn repository. Fostering this "don't commit until it's right" mentality is a natural consequence of not wanting to look like an idiot to your colleagues, and works in perfectly fine with the svn-based workflow.

but git doesn't work that way!

Or should I say that git doesn't *need* to work that way. After all, you still need to make sure that your tests pass and don't break on the CI server... but what I've found is that you need to get over the whole git commit thing. It may be named the same thing as svn commit - but it doesn't mean the same consequences (eg that your colleagues will all see that the feature's only half-complete and the tests are all spazzing out).

Instead, change the way you think: the command that *matters* now is actually git push. You can commit whatever you like to your local repository, even if it breaks the build; it's only when you push up to the remote repo that it must be working perfectly.

Any other problems with this?

Unfortunately, there are some other consequences to this small change in workflow. One of them being the fact that you can't do a 'git diff' that covers all your changes since last push. git status and git diff are *just* like svn status and svn diff - they check against the latest commit, not the latest 'push to remote', which means it's hard to do a complete check of all your changes before going 'live'... you have to just trust that all your smaller commits all add up to the Right Thing.

That makes me feel uncomfortable as I like to be sure. I know about human error - and I know that I'm as prone to it as the next guy...

Having to make a patch-against master and then read through *that* (which is far less clear to read than a diff) is not a good substitute, IMO. If anybody has a good way on how to mutate the workflow to accommodate this I'd love to know.

a new workflow?

I'm still working on this but so far I've got:

1. Clone a fresh, clean copy of the repository (if I don't already have one)
2. git checkout -b my_new_branch
3. add tests and/or make some changes
4. do a git diff and check this change - eyeball it to make sure there's nothing obviously wrong
5. git commit (then repeatedly return to step 3 until 'done')
6. check the rest of the tests run - make changes (and git commit) until 'done done'
7. do a git pull origin master to pull in latest changes by others and make sure the tests *still* pass
8. fix any merge-conflicts and commit the merge
9. git push

This is still a work-in-progress, and I would appreciate informed opinions, advice or your own war stories.

Notes:
[1] IMO the git developers could learn a thing or two from Kathy Sierra... but that's another topic.[4]
[2] If you've ever got into a state where you can't run git commit because you're in a 'failed merge', you can't git pull because you get 'fatal: merging of trees' or 'Automatic merge failed; fix conflicts and then commit the result.'. You edit the files to un-conflict them and try to reapply your stash you suddenly get 'CONFLICT (content): Merge conflict in ...' again... After thrashing around for a while between git stash, git pull, updating merged files then trying to re-apply your stash before git committing... I can tell you where I wanted to stick git.
[3] If you're anything like me, you look on the words git reset --hard HEAD with some trepidation. You just can't quite believe that blowing everything away in your working copy is the only way out of a simple merge-conflict.
[4]...and please don't just tell me that git is open-source and I should just go hack on git myself if I hate it so much. In theory I absolutely agree with you, but in practice I can only work on one thing at a time - and right now I'm still working on Active Resource, some projects of my own, a novel...

How to monkey patch a gem

Is a gem you're using missing something small - something easy to fix?
You've made a patch and submitted it, but they just aren't responding?

I've found two ways to monkey patch it in place until they get around to pulling your patch in. They both have their pros and cons.

1) vendor the gem in place and hack

Do this if you give up on the gem people ever caring about your change (maybe the gem's been abandoned), if you're only using the gem in a single application; or the patch is only relevant to your one specific application; or if you want to put your changes into the repository for your application.

How:

1. rake gem:unpack into vendor/gems
2. sudo gem uninstall the gem from system if not in use for other apps
3. add the gem code into the repository
4. make your patch in place
5. update the version (eg use jeweller and bump version or hand-hack the gempsec and rename the directory)
6. add a config.gem declaration and depend on your version of the gem OR add a line to your Gemfile - and use vendored_at to point at it

Pros:

1. you can keep track of what changes you've made to the gem in the same place as your application directory - thus it's good for changes that are very specific to your own application-code (ie aren't really relevant or shareable with the wider community or your other apps)
2. it's pretty easy for a quick patch that you know is going to be pulled into the main gem shortly. It's easy to blow away the vendored version once the 'real' version is ready.

Cons:

1. if you're not using gem bundle yet, it's annoying to get your application to use your custom gem
2. it's not easily shareable between your applications if it's hidden in the vendor directory of only one - you may need some complicated extra-directory + symlinking to work...
3. if the gem is ever updated upstream, you have to do nasty things to get the new version (hint: before upgrading, make a patch via your source control... then blow away the gem directory... then download the new gem version... then reapply your patch). :P

2) fork the github repo

If the gem is on github, you can fork the gem there - this is especially good if you're going to reuse your patched gem in multiple applications, and/or make your patches available.

How:

1. Fork the gem OR create a github repo for the gem and push the code up there OR clone locally and create your own repo for it
2. Make your changes, and commit them to the repository as usual
3. In config.gem use :source => 'git::github.org/me/gemname' or gem bundle's Gemfile use :git => 'github.org/me/gemname' (or appropriate location)
4. optionally: be nice and make a pull-request to the original repo

Pros:

1. can easily pull changes from the upstream repository and integrate with your own patches
2. good for sharing amongst multiple of your own applications
3. makes your changes available to other developers that may be facing the same problem
4. good for when the main gem is not longer under development (or only sporadically updated... or has developers that don't agree with your changes)

Cons:

1. more work than a quick hack in vendor
2. must be tracked separately to your own code
3. you might not want to host outside of your own system (of course, nothing wrong with cloning then pushing to your own local repo, rather than github)

Conclusions?

We had a couple of the former and began to run into the issues stated. We discovered, of course, that quick hacks tend to turn into longer, more lasting changes so found that might as well have just done it 'properly' the first time and are now moving towards the latter solution - even setting up our own git-server for the gems we don't want to release back into the wild. YMMV :)

gem bundle with capistrano

[Update: There's a capistrano recipe in bundler now, if you just require "bundler/capistrano" apparently it automatically runs bundler install and installs the gems in shared/vendor_bundle]

Ok, so now you've converted your existing rails project to using gem bundle, and it runs fine on your development box, how do you update your capistrano recipes to do it automatically?

Current wisdom is varied. You can:

1. check *everything* into your source control (very heavy!)
2. check in only your Gemfile and the vendor/bundler_gems/cache (which holds all the *.gem files) into source control (still pretty heavy, but makes sure you don't need internet access every time you deploy - use 'gem bundle --cached' in your recipes
3. symlink your vendor/bundler_gems/cache directory to a shared location, and then all you need to checkin is the Gemfile (quicker, still needs to unpack/bundle gems each time)
4. symlink all your vendor/bundler_gems/ directories to a shared location, and then all you need to checkin is the Gemfile (quick and not too heavy)
5. symlink the entire vendor/bundler_gems directory to a shared location (much quicker).

5: Symlink the lot!

This would be my preference. For a single project deployed on production, there should be no reason why we can't just symlink the whole bundled-gem directory, and let the Gemfile keep that directory up-to-date. This feels no different to using a shared file-attachments directory.

Sadly doesn't work due to this:
No such file or directory - MyProjectRailsRoot/vendor/bundler_gems/../../Gemfile
which will resolve to two directories above the *shared symlinked* directory :P

This is because the bundled_gems generates an 'environment.rb' that points back up the directory at the Gemfile that created it... by using a relative path that unfortunately hits the symlink as described above. It'd be nice to be able to tell gem_bundler to make that link absolute...

If anyone knows a way around this, please do let me know!

So we fall back on 3 or 4. My first attempt was to use 3:

3: ci the Gemfile, symlink the cache directory

This seems to be a reasonable compromise. Our deployments are getting pretty damn bloated as it is - with all those plugins and vendored rails etc etc... we don't want to add anything more if we can avoid it. Even gems. We have had no problem with downloading gemfiles, so there's no need to check them in, as long as the deployed directory has access to them when needed. Thus, checking the Gemfile, symlink the cache directory... maybe even symlink, the 'gems' directory sometime too.

So, how to do it?

First - prepare the bundler_gem directory for the first time login to your deployment server, and create the shared directory in, eg:

mkdir ../shared/bundler_gems 
mkdir ../shared/bundler_gems/cache

Next add the symlink and gem bundle commands to your capistrano deployment recipe:

# symlink to the shared gem-cache path then bundle our gems
namespace :gems do
  task :bundle, :roles => :app do
    run "cd #{release_path} && ln -nfs #{shared_path}/bundler_gems/cache #{release_path}/vendor/bundler_gems/cache && gem bundle"
  end
end

after 'deploy:update_code', 'gems:bundle'

And you *should* be able to deploy fine from there. My experience wasn't completely smooth, and I had to set up on the server manually the first time - but from then on the capistrano-task worked fine. Would love to hear your own experiences...

Hudson

If you're using hudson for Continuous integration, you can achieve the same effect by updating the Hudson build script adding:

# prepare gems and shared gem-cache path
ln -nfs <hudson config path>/bundler_gems/cache  vendor/bundler_gems/cache
gem bundle

Also - if you have test/development-environment-only gems... make sure you add your integration-server's environment to the Gemfile environment list, or Hudson won't be able to find, say, rcov or rspec - and then the build will break.

4: ci the Gemfile, symlink the sub-directories

I started with the solution described above, but it still takes a while unpacking all the gems. I'd much prefer to use the solution #5, but that fails due to the relative links in environment.rb. So the ugly compromise is to symlink everything *except* the environment.rb

It works and we can deploy ok with capistrano... but I'm looking for a better solution.

...and after a few deploys...

Well, it was a nice try, but after a few deploys suddenly we started getting breakages... the application couldn't find shoulda for some reason. Now we use a hacked version of shoulda and have vendored it as that's a quickie way to monkey patch a gem.

We told gem bundle where we'd vendored it, and it all seemed to work fine. Unfortunately it broke, and a quick look at the symlinked 'gems' directory tells us why:

rails-2.3.4 -> /var/www/apps/my_app/releases/20091201120443/vendor/rails/railties/
shoulda-2.10.2 -> /var/www/apps/my_app/releases/20091201120443/vendor/bundler_gems/dirs/shoulda

What you can't see are these lines blinking red. What you can see are that these gems are pointing at a specific revision's vendor directory... and not the latest one! Coupled with a :keep_releases = > 4... that release's directory is quite likely to disappear very quickly - and in this case, it already has. This makes these symlinks (set up by gem bundle during release 20091201120443) point to a vendor directory that no longer exists. So it's really not as much of a surprise that our app can't find shoulda anymore.

I think the problem comes along because of gem bundle's rather useful feature of not reloading a gem if it's already installed. It looks to see if the specification exists - if so, it assumes that it's been installed correctly - it doesn't check that the vendored location still exists. That unfortunately spells our doom when capistrano deploys: because gem bundle runs from the newly-created release-directory, that's where the symlink is initially set up. gem bundle doesn't then check it later - even though later its been swept away in a release-cleanup.

So - we're currently working on a fix. Two options present themselves:
1) find a way for gem bundle to symlink from 'releases/current'. This means it has to exist before we do a gem bundle... and that's dangerous because it lets users through into a not-yet-working release. Or
2) we could not vendor any gems - but set up our own gem-server. This will work, but a bit more of a hassle than we prefer for vendored gems.

Troubleshooting:

Git repository 'git://github.com/taryneast/shoulda.git' has not been cloned yet

The recipes online all tell you to use gem bundle --cached and that's a great idea if network connectivity is scarce. As it uses gems that are already in a cache (without going to fetch them remotely)... but it will fail if you don't already have the gems in the cache. SO it relies on you applying solution number 2 above.

There are two solutions:
The first is to just use gem bundle always in your recipes.
The other is to use gem bundle on your development directory - then check the cache into your source control instead of symlinking it to a shared directory. (ie use solution 2 above). This will work just fine, and if you add a new gem, you'll have to make sure you run gem bundle on your working directory before checking in.

I'm not sure of a nice way to get around this if you're using solution 3. It might be worth setting up a rake task that checks if the Gemfile changed and opportunistically runs gem bundle instead of the --cached version (in fact, it'd be nice if gem bundle --cached had a --fetch-if-not-present option!). If you have a solution that worked for you, let me know.

rake aborted! no such file to load -- spec/rake/spectask

I got this when running rcov. It just means you need to add a few extras gems to your Gemfile. We needed all of these. YMMV

  only [:development, :test, :integration] do
    gem 'rcov'  # for test coverage reports
    gem 'rspec' # rcov needs this...
    gem 'ci_reporter' # used somehow by rake+rcov
    gem 'ruby-prof' # used by Hudson for profiling
  end

:bundle => false with disable_system_gems

Looks like these two don't play nice. ie if you choose for a single gem to be unbundled - but have disable_system_gems set - it isn't smart enough to realise that this one gem is meant to be an exception to the 'disable' rule. If you have any unbundled gems, make sure you remove disable_system_gems - or your application simply won't find it.

Convert from config.gem to gem bundler

Why gem bundler?

Our sysadmin hates rake gems:install

It seems to work for me, but all sorts of mayhem ensues when he tries to run it on production... of course I have a sneaking suspicion it's really our fault. After all - we tend to forget that we already happened to globally-install some gem while we were just playing around with it... which means we didn't bother putting it into the config.gem section of environment.rb... oops!

However, there's a new option on the horizon that looks pretty interesting, and is built to sort out some of the uglier issues involved in gem-herding: gem bundler

Yehuda has written a very thorough a tutorial on how to set up gem bundler. But I find it kinda mixes rails and non-rails instructions and it's not so clear on where some things go. I found it a little fiddly to figure out. So here's the step-by step instructions that I used to convert an existing Rails 2.3 project.

Step-by-step instructions

1: Install bundler

sudo gem install bundler

2: create the preinitializer.rb file

Yehuda gave some code in his tutorial that will load everything in the right order. You only need to copy/paste it once and it will then Just Work.

Go grab the code from his tutorial (about halfway down the page) and save it in a file: config/preinitializer.rb

Don't forget to add that file to your source control!

Update: If you're using Passenger, update the code to use:

module Rails
  class Boot
  #...
  end
end

Instead of:

class Rails::Boot
  #...
end

3. create the new gems directory

mkdir vendor/bundler_gems

Add this directory to your source control now - while it's still empty!

While you're at it, open up your source-control's ignore file (eg .gitignore) and add the following:

vendor/bundler_gems/gems
vendor/bundler_gems/specifications
vendor/bundler_gems/doc
vendor/bundler_gems/environment.rb

4. Create the Gemfile

Edit a file called Gemfile in the rails-root of your application (ie right next to your Rakefile)

At the top, add these lines (comments optional):

# because rails is precious about vendor/gems
bundle_path "vendor/bundler_gems"
# this line forces us to use only the bundled gems - making it safer to
# deploy knowing that we won't accidentally assume a gem in existence
# somewhere in the wider world.
disable_system_gems

Again: don't forget to add the Gemfile to your source control!

5. Add your gems to the Gemfile

Now comes the big step - you must copy all your config.gem settings from environment.rb to Gemfile. You can do this almost completely wholesale. For each line, remove the config. from the beginning and then, if they have a version number, remove the :version => and just put the number as the second param. I think an example is in order, so the following line:
config.gem 'rubyist-aasm', :version => '2.1.1', :lib => 'aasm'
becomes:
gem 'rubyist-aasm', '2.1.1', :require => 'aasm'

For most simple gem config lines, this should be enough so that they Just Work. For more complicated config.gem dependencies, refer to the Gemfile documentation in Yehuda's tutorial.

If you already have gems in vendor/gems You can specify that bundler uses them - but you have to be specific about the directory. eg:
gem 'my_cool_gem', '2.1.1', :path => 'vendor/gems/my_cool_gem-2.1.1'

Extra bonus: if you have gems that are *only* important for, say, your test environments, you can add special 'only' and 'except' instructions (or whole sections!) that are environment-specific and keep your staging/production environments gem-free eg:

gem "sqlite3-ruby", :require => "sqlite3" , :only => :development
only :cucumber do
  gem 'cucumber'
  gem 'selenium-client', :require => 'selenium/client'
end
except [:staging, :production] do
  gem 'mocha'          # http://mocha.rubyforge.org/
  gem "redgreen"       # makes your test output pretty!
  gem "rcov"
  gem 'rspec'
end

5a: Add rails

Now... at the top of the Gemfile, add:
gem 'rails', '2.3.4'
(or whatever version you currently use)... otherwise your rails app won't do much! :)

Obviously - if you've vendored rails you will need to specify that in the Gemfile way eg:
gem 'rails', '2.3.4', :path => 'vendor/rails/railties'

If you've opted *not* to disable_system_gems, you won't need this line at all. Alternatively, you could tell the Gemfile to use the system-version anyway thus:
gem 'rails', '2.3.4', :bundle => false

Also, I'd recommend adding the following too:

 gem 'ruby-debug', :except => 'production'  # used by active-support!

6. Let Rails/Rake find your gems:

Edit 'config/environment.rb' and at the bottom (just immediately after the Rails::Initializer.run block, add:

# This is for gem-bundler to find all our gems
require 'vendor/bundler_gems/environment.rb' # add dependenceies to load paths
Bundler.require_env :optional_environment    # actually require the files

7. Give it a whirl

From the rails root directory run gem bundle

The bundler should tell you that it is resolving dependencies, then downloading and installing the gems. You can watch them appearing in the bundler_gems/cache directory :)

and you're done!

...unless something fails and it can't find one - which means you probably forgot to add it to config.gems in the first place! ;)

PS: ok... so I've also noticed you sometimes have to specify gems that your plugins use too - so it may not be entirely your fault... ;)

PPS: if Rake can't find your bundled gems - check that config/preinitializer.rb is set up correctly!

shoulda should play nice with Il8n errors

...but it doesn't.

It took me a while to figure out what was going wrong with it, but I discovered that ti was checking against the literal locale-based string (eg "Please enter a value for {{attribute}}") rather than the string that the error would really show up as (eg "Please enter a value for Name").

I've put a fix into my shoulda branch on github along with a new macro: should_ensure_length_at_most (the opposite of the existing should_ensure_length_at_least). If you wanna just patch an existing system, I've put up a pastie of my patch here.