Software: no compensation necessary

I've just been reading "The World Is Flat" by Thomas L Friedman. It's an interesting book with lots good to say[1] about how the world is rapidly flattening - ie everywhere is becoming reachable from anywhere.

Along the way (and only incidentally to the main point) the author brings up the concept of Open Source Software.

He discusses the concept in a fairly positive light. He seems quite fascinated by the phenomenon - though doesn't seem to really get why people would do it. Though he makes a valiant effort to see things from the POV of the average OSS contributor, it's still fairly clear that he thinks it's a bit of a quaint, idealistic example of altruism (recognising the "pay it forward" aspect), rather than an effective and useful business technique. This is exemplified by the fact that he interviews a Microsoft Manager to ask whether or not it is sustainable as a business concept. Not bothering with an Open Source advocate, just an employee of one of the biggest proprietary systems out there.

One point stands out from what this employee says about OSS. He states that money is the main motivator for building software. From this the author concludes that any "really good software" will more likely be built by big software houses (eg Microsoft) than by OSS initiatives... after all, they have a lot of money, that can buy a lot of development time. He therefore concludes that The big software houses will always build the best software - simply because they can sustainably pay the most developers.

It's an interesting theory, and the most logical for those who don't really understand the motivation of the average OSS contributor. The reasoning behind this conclusion is of itself quite interesting. They claim that software is developed because businesses need stuff to be done - and thus they pay developers to do it. Nothing wrong with that... but they seem to mistake the effect for the cause and assume that the motivator for software to be built - is the money.

This is not the case.

Software is susceptible to the old phrase of "Necessity is the mother of invention" - ie the cause of software to be built is the business's need for it. The money is a secondary affector - ie the need causes the business to find money to make it happen. The originating need will be present regardless of whether the business has money to pay for developers... after all, even if the business has no money, the need will still be there for the software.

If the business happens to be made up of developers, then they are quite capable of creating the software, regardless of payment... and that is how you get startups: built and run by developers who see the need but don't necessarily have money to pay other developers - they just need the software to get their own business running, and are ready and willing to work in their own spare time to make it happen (nothing wrong with Sweat Equity).

The Open Source world calls this motivation "scratching your own itch" and it is recognised as the primary motivator for Open Source contribution[2]. Even for the really big projects.

Linux, for example, was Linus Torvalds' project to build himself an OS that didn't suck. Since he released it, thousands of other individuals have contributed to it for the same reason. It didn't do what they needed, so rather than bitching about it, they put their money where their mouth was and wrote the fix themselves... contributing it back to the OSS community... for no compensation except the "pay it forward" principle.

Which brings me to the other book I've just finished reading: "Drive" by Daniel Pink. One of the principle findings of this book is that fiscal compensation actually gets in the way of intrinsic motivation[3]. You can incentivise a developer to build something, but then they dissociate their ownership of the end product to the person doing the incentivising... care factor drops radically and you end up with... well... Microsoft products. Corporate and cold, buggy and with no real way to guarantee that the problems will ever get fixed.

This difference in motivation easily accounts for the fact that MS products are made primarily for creating more money for Microsoft (for which purpose they perform admirably), but only a passing interest in actually being fit for use: ie only so much as they can get away with before too many people stop purchasing them.

Now, there's nothing inherently wrong with that. More power to them for staying alive so long, and rather profitably - by keeping so many people satisfied for so long with what they're prepared to shell out... but there's a reason that I haven't used Microsoft products for over a decade now...

By comparison, the OSS world is driven purely by user requirements. If software is not fit for purpose, it simply won't be used... but more importantly, because there's nobody paying for it - if it doesn't do what it's supposed to, it will never be born

Conversely, if it is built, you can guarantee that it meets the requirements of at least *one* real customer (unlike many products built purely with potential revenue as a motivator).

OSS lives and dies on it's own merits... no compensation necessary.

Notes:

[1] Though I'd take a lot of it with a pinch of salt as well. This guy is clearly of the "yay capitalism, it solves all the world's ills" school of thought... though is slightly tempered by realising that humans are human, and some of them actually need supporting by those who are better-off... it's an interesting act of comparison to read this side-by-side with "The Shock Doctrine" by Naomi Klein - just as equally biased in the opposite direction, but also an fascinating read. I think if I ever put these two books together on a shelf, they'd annihilate one another with a very loud bang.

[2] If you're interested in this topic, "The Cathedral & the Bazaar" by Eric S. Raymond is an excellent book that explains the difference between what can be built from the ground up by need-motivated developers rather than the top-down software built by big software houses spending money to pay for what they want.

[3] which can be paraphrased as that innate curiosity and enjoyment of a challenge that occurs when you take on a new puzzle

Email campaign tracking with Silverpop

SilverPop offer themselves to the world as an "Engagement marketing solution". They provide click-stream and conversion tracking based on your email marketing campaigns, by putting unique identifiers in the links of your mailings (which they can send out in bulk). If you store these IDs and ping them back to SilverPop's click-stream and conversion-tracking servelets, this means you can get complete conversion analysis based on your email marketing campaigns.

I'm currently working for Moneyspyder, and one of our clients has signed up for a Silverpop account - which means we need a way to easily integrate with SilverPop - and the examples they give are all in php (and a little less than dynamic). So here's my Ruby-on-Rails solution.

1) Cookify the IDs

When a customer clicks on a link in one of the SilverPop emails - Silverpop has cleverly adding tracking IDs to identify the mailing-job and individual customer. When we send tracking information back to SilverPop - we need to send back these ids so the click-tracking is stored against the correct campaign and individual.

Of course, the parameters will disappear after the first time the user clicks another link, and we need to persist the data - to keep tracking all the pages they click on until they eventually convert!

The easiest way to keep track of this to stick it all into the session - then we can just check if it's there and send it each time we want to ping to SilverPop... and of course the best place to do session-wrangling is in a before_filter. So stick the following into something like application.rb

  before_filter :save_silverpop_data_in_session

  # This method does the storing of the ids from the URL into a session
  # cookie for later sending.
  # It will replace any existing values in the cookie - which represents the
  # user having returned to the site after viewing (and clicking on) another
  # link from a different email campaign.
  def save_silverpop_data_in_session
    silverpop_params = %w{spmailingid spuserid spjobid spreportid}
    if silverpop_params.all? {|f| params.keys.map{|k|k.to_s.downcase}.include?(f) }
      params_down = {} # ugly but necessary to get past browser case insensitivity issues
      params.each_pair {|k,v| params_down[:"#{k.to_s.downcase}"] = params[k]  if silverpop_params.include?(k.to_s.downcase) }
      session[:silverpop] = {:m => params_down[:spmailingid], :r => params_down[:spuserid],
                             :j => params_down[:spjobid], :rj => params_down[:spreportid]}
    end

  end

2) Configure your pod

SilverPop calls it's servers "pods" - and you could be using any one of them. This works better as a configuration option than hard-coded in your code - and lets you set up a link to the test-server on your dev/test environments. but in environment.rb you'll have something like this:

  # SilverPop URL = mailing list manager/ClickStream Analysis
  SILVERPOP_SITE_URL =  "recp.mkt41.net"
  SILVERPOP_SITE_HTTPS_URL =  "marketer4.silverpop.com"

3) Helping hands

Next up is to figure out how to ping SilverPop... which they helpfully give us an img tag example of how to build up the correct url with all the requisite values.

SilverPop has two servelets - one for accepting pings for click-stream tracking and one for the conversions. But they're almost identical, just taking different parameters... and I'm lazy and don't want to have to remember all the common details of how to do this in each place in the site. Thus: helpers to the rescue!

  # This method generates the code that calls the ClickStream tracking
  # servelet - we pass in the page name and page URL, along with the values
  # passed through from the last silverpop email - saved in the session
  def silverpop_click_stream_ping(page_name, page_url)
    silverpop_link('cst', :name => page_name, :s => page_url)
  end


  # This method generates the code that calls the Conversion Tracking
  # servelet - we pass in the "action", "detail" and "value" flags - 
  # eg "silverpop_conversion_ping 'CompletedOrder', order.id, order.grand_total" 
  # along with the values passed through from the last silverpop email -
  # saved in the session
  def silverpop_conversion_ping(action,detail,value = nil)
    silverpop_link('cot', :a => action, :d => detail, :amt => value)
  end


  # generate the SilverPop ping-image based on required servelet and parameters
  def silverpop_link(servelet, options)
    # skip out early if this user hasn't come through a silverpop email
    return nil unless session[:silverpop].present?

    base_url = (/https/ =~ request.protocol) ?  "https://#{SILVERPOP_SITE_HTTPS_URL}" : "http://#{SILVERPOP_SITE_URL}" 

    image_tag "#{base_url}/#{servelet}?#{session[:silverpop].merge(options).to_query}", 
      :height => 1, :width => 1, :alt => ""
    # we'd like this alt-tag: "Silverpop #{servelet.upcase} Servelet Ping"
    # to be accessible... but the above is not *really* an image, and thus will display 
    # the alt-text in the html at present... this sux, but I have yet to talk to silverpop 
    # about a way around this.
  end

4) Click-stream - analyse!

So, now it's time to get down and dirty with the click-stream analysis. This couldn't be more simple. Just use the helper to pop a link into your main layout. eg:

  <!-- silverpop cst ping -->>
  <%= silverpop_click_stream_ping(@page_title, url_for(:only_path => false)) -%>

Now, as you can see, this mainly works by using our dynamic page-title and a link to the current-page (using the empty url_for trick). Note that if you leave off the "only_path=false" option it won't provide the hostname. This may be what you want if you want to roll up multiple mirrored domains. You'll also have to adjust the page-name parameter as necessary for the way you generate the page-title... but otherwise you're good to go and this means every page-click gets tracked back to SilverPop from now on.

With one caveat... if you have AJAX-updating, you may need to figure out a neato trick of putting the ping-link into the newly-generated page-pieces or these "clicks" won't get tracked as the layout won't see them as new pages. I'll leave that as an exercise for the reader as it's very site-specific.

5) Mine your conversion gold

Now only the final, and most important, part is left - tracking actual conversions.

A lot has already been written about what constitutes an important conversion for your site. I won't repeat it all here as you can track heaps, and it's really down to what is important for your business. So I'll pretend we only care about when a customer completes an order - which we know because they land on the "thank you" page.

Which means we need to pop a link to the COT servelet there and pass in the important details... nothing easier:

  <!-- silverpop cot ping -->>
  <%= silverpop_conversion_ping("Order Complete", @order.id, @order.grand_total) -%>

Now SilverPop will monitor our marketing mails from go to whoa - and even know which order they completed and, most important, how much money they ended up giving us... Gold!

Update: Sorry for the repost (anyone that noticed). I fixed a few bugs after I finally got a chance to fully test this out.

Anybody implementing a Silverpop solution should also be aware that Silverpop's servers (at this moment) do not correctly redirect links from mailings sent from their test server. Instead you get redirected to your website, but without any values in the required query parameters (eg spMailingID etc are empty).

To do an end-to-end test of SilverPop's link redirection, you must create a real newsletter mailing on the live server, and just restrict the recipients (eg send it to yourself and any other devs on your team).

Stored Procedure-fu

So, I've been tuning my db performance on our heaviest database call... and this involved adding a quickie stored procedure to make life a little simpler.

It's nothing complex, just does a really simple distance calculation (using pythagorus)... which we needed because most versions of MySql don't come fully-loaded with all those nice functions that you can find in the manual.

Now the problem with stored procedures, is how to add them to the db without too much hassle. I have heard there are gems out there - but they seem a little like overkill, and the ones I've seen have been "build it on the fly"... I just want to put it into the db and leave it there.

So what's the quickest and easiest way?

A migration:

class AddStoredProcedures < ActiveRecord::Migration
  def self.up
    # function distance(lat1,lng1,lat2,lng2)
    #   sqrt((lat2 - lat1)^2 + (lng2 - lng1)^2)
    # end
    execute <<EOS 
      CREATE FUNCTION Distance(lat1 FLOAT, lng1 FLOAT, lat2 FLOAT, lng2 FLOAT) 
      RETURNS FLOAT
      DETERMINISTIC
      BEGIN
        DECLARE dist FLOAT;
        SET dist = Sqrt(Pow((lat2 - lat1),2) + Pow((lng2 - lng1),2));
        RETURN dist;
      END
EOS
  end

  def self.down
    execute "DROP FUNCTION IF EXISTS `Distance`" 
  end
end

Down sides?

I'm told that using migrations for stored procedures has one major drawback - that if you want to change the procedure later then it's annoying to maintain - and can be the wrong version etc... In this case that's not a problem. Pythagorus isn't likely the change any time soon, so I'm unlikely to need to update it.

But there is one annoyance about putting it as a migration, and that's the fact that it doesn't show up as part of the schema... which means our Test environment can't find it. :P

This is a bit annoying, and I haven't found an elegant solution to the problem, but to get the test suite running, I have found a hack...

Re-run the migration:

class ActiveSupport::TestCase
  # ... lots of guff

  # Run stored procedures to add to the test db
  require File.expand_path(File.dirname(__FILE__) + "/../db/migrate/20100528190204_add_stored_procedures.rb")
  AddStoredProcedures.down # clear it out first, to be sure
  AddStoredProcedures.up
end

It's ugly, but it works. You may note I have to run the "down" then the "up" - this is because ActiveSupport ::TestCase actually gets loaded for each type of test-case you are running (eg once for Unit tests and again for Functional etc) and the "up" barfs if you try to create a procedure that has already been created in the db.

Otherwise, it runs and lets you get on with your work...

if anybody has a more elegant, but unheavy solution to the whole issue of Stored Procedures in test dbs... please do let me know.

Blogger has a new template editor...

...I couldn't resist.

So my blog is now more ruby-ish. :)

PS: if you use the template editor... don't forget to add back stuff you customised in the template (eg your google analytics widget) *blush*

Alternatively make them into page objects so they aren't a part of the template at all

How *NOT* to do customer service

1. Start by telling your customer the work is not good enough. That the resolution will make it pixellated and that will look terrible.
2. Then say that the image has "keylines" that have to be removed and "crop lines" added "outside the trim area" - without explaining what they are
3. When your customer naiively asks what these mean, instead of explaining it - tell her that you haven't yet received the artwork because the only files they received don't have these things
4. When the customer says she sent files through and asks what is needed to change them into files that will meet their requirements (giving a detailed guess at what that might be), tell her again that you need these things - without explaining exactly what is needed or confirming that what she suggested is actually what you needed - even if she's later proved correct. Tell her that she is running out of time to get this done...
5. When your customer calls, make sure your receptionist (or colleague) answers, and that she tells your customer that you are with some other important client... and get *her* to also tell your customer that the artwork isn't good enough and go on at length about how it's such low resolution that it'll look all pixellated and low quality that it'll look terrible. Make sure she promises that you'll call her back before close of business.
6. When your customer has received no calls or emails and it's near close of business... and calls you. Answer immediately and...
   1. Begin by telling her off again at the low quality of the images (due to the resolution).
   2. When she mentions she thought you'd be calling her back, tell her you have *so* many other clients and you're very busy, you know...
   3. When she asks if you'll please just explain what she needs to do to add the crop lines - don't explain, tell her that you were expecting to speak with a designer who should already know these things.
   4. When she suggests that not all clients know the industry jargon of printing businesses and again asks for you to explain what to do, tell her to go and look up "crop lines" in the dictionary
   5. When she says that sometimes in your industry you have to deal with people that aren't specialists (she gives an example that if you'd asked her to set up a website, she wouldn't expect you to know what Rails, MySQL or AJAX were)... then tell her that you simply don't have time to explain to your customers how to add crop lines to an image, and that you only deal with designers that already know what crop lines are...

background...

We're launching our new startup this weekend (ie tomorrow) and needed new business cards ASAP. So my colleague called around the local while-u-wait business card dealers and found one that was willing to print and ship in time for tomorrow.

I'd quickly thrown together some business card images that were the exact size/shape required. They aren't brilliant, but they'll do for the weekend after which we can get some better ones done up.

I have a day-job... so while I'd love to go out and find out all about crop lines and trim-borders and whatever... I need to actually spend the day making money for my employer...

When I got off the final phone call and calmed down for a few minutes, I managed to find some time to go look it up. Apparently, all I need to do is exactly what I suggested in the reply to the second email - ie, expand the image size and add borders a little larger than where the old image-shape used to be... this would have taken him all of one minute to explain - just as it did me just now.

I recommend that you *don't* purchase from Kall Kwik 781 in Eton Street, Richmond