Tuesday, 28 April 2009

Playing nice with XML and HTTP

So we've got a setup that has a remote API that we're accessing using HyperactiveResource (an extended version of ActiveResource). Now, I'm using Rails to simulate the remote API (for the purposes of testing), and I've come across some annoying behaviour.

One issue is that standard rails routing for a RESTful interface will direct a badly-constructed (or non-existent) URL to a real action... let me demonstrate thus:

Real member path: /users/1.xml Routed to: :controller => 'users', :action => 'show', :id => '1'
Real named collection path: /users/count.xml Routed to: :controller => 'users', :action => 'count'
Non-existent path: /users_party_on.xml Routed to: "Bad Request" handler
Non-existent path2: /users/party_on.xml Routed to: :controller => 'users', :action => 'show', :id => 'party_on'

If I called the last URL with curl, I'd expect to be routed to the "Bad request" handler and receive some sort of error-like http-status and an XML message explaining that no route exists or something similar... what I get instead is a horrible big *html* page telling me it couldn't find the user with an id of "party_on" (unsurprisingly).

So what do I want to have happen? I'd rather this stuff was caught in the router. It'd be nice if there were a way to tell the router that your :controller/:action/:id is only valid for a certain formatting of the :id field. If anybody out there on teh Intarwebs knows how to do that, please tell me now!

Unfortunately, it doesn't seem to do this... and in any case, the router/dispatcher also doesn't seem to return XML to an XML-request... it only seems to know how to handle HTML-based errors (by spitting back the public error pages[1]).

So instead, what I need is to return a "URL not found"-style xml error at the appropriate time.

Most of my controllers have a "find_" function on the member-functions (ie just @thing = Thing.find(paramd[:id])). Now, since the bad URLs tend to converge on the "show" action - this seems as good a place to put a bad-request filter as any. I'll also incorporate it with the 404-code that also seems missing when a doesn't exist (or is not accessible by this person).

So this calls for a helper-method as below, as a hack to fix this lack of proper routing.

  # convenience method for extracting the expected model name from the
  # controller name
  # Note: expects the model to be rails-standard eg "ThingsController"
  # should map to the Thing model
  def model_name
    self.controller_name.singularize.camelize
  end

  # use this to skip out early and return better http status codes for XML
  # requests.
  #
  def find_thing
    the_id = params[:id]
    # ids should be numeric. If they're not - we accidentally got through
    # the router with an unrecognised action - because Railsy named-routes
    # that *don't* exist, look like the "show" action with a bad id.
    if !the_id.blank? && !the_id.to_i.is_a?(Numeric)
      # skip out with a 400 early...
      respond_to do |format|
        format.xml do
          return render :xml => 'Error: URL not recognised', :status => :bad_request 
        end
      end
    end
    begin
      thing = model_name.constantize.find(the_id.to_i)
    rescue ActiveRecord::RecordNotFound
      # skip out with a 404 early...
      respond_to do |format|
        format.xml do
          return render :xml => 'Error: resource not found', :status => :not_found 
        end
      end
    end
    thing
  end

Due to how controller before_filters work[2], you should use this code thus:

class UsersController < ApplicationController
   before_filter :find_user, :except => [:new, :create, :index, :count]
   # actions all go here
   # ...
protected
  def find_user
    @user = find_thing
  end
end

Caveat: if you have non-standard naming of models/controller - the model_name.constantize will not work... so you may want to modify this to pass in an optional klass param.

Notes:
[1] and when exactly are they going to make these into templates so we can use the standard layout rather than hand-coding it for each one?
[2] ie, I'm too slack to figure out exactly how to do: @thingy = in a block passed to the before_filter command. Again - if you know how, let me know.

Monday, 27 April 2009

Testing ActiveResource

I've been banging my head against the wall that is ActiveResource for a while now. One big problem is actually getting the testing correct.

Take 1: HTTPMock

The supposedly sanctioned expectation is that you test it against the provided HTTPmock... unfortunately, this is great for testing that ActiveResource makes the correct remote calls eg, when you update a user that ARes POSTs to /users/123.xml... and that it reacts to a (pre-stubbed) 404 by raising a ResourceNotFound; but it doesn't allow you to actually test your model - eg to assert that when you update your user's login field and reload your model, that your user can now login with the new details...

AFAICS you can't test anything that is actually meaningful or useful to your business logic - which kinda defeats the point IMO.

Even with a lot of hacking, I couldn't get the kind of dynamic mock backend that I needed for any meaningful test-coverage.

Take 2: Mocking with Stump

Next up was to try to mock/stub out the backend functions appropriately. I'd heard some good stuff about the stump plugin and dutifully installed it to have a go.

Stump let me do much more dynamic testing. I could stub out the "create" and "find" functions and make it respond with a mock remote object that I could then further stub out the "save" and "update" functions... and after a while it seemed like I was stubbing out functions to return stubbed out stubs and it all got a little bit circular... and in many cases: really complex.

I also found that I got the code to work in the browser I'd often still have to spend ages getting the tests to work with the right set of mocks and stubs to patch all the ways that rails could leak out and try to call the "real" API. :P

I also noticed a few times where I'd somehow accidentally plastered over a bug with a stub by assuming the return-value. I only noticed this because the tests would run fine - yet trying the same operation in the browser revealed the bug.

Now how can I test if I can't rely on the test to actually test? My safety net suddenly has big gaping holes in it that I'll only find if I fall through them while testing by hand! This is such a step backwards that it's horrifying!

Then I hit a wall.

We have a user and during creation (and update) we need to check that the login is unique - which requires a find on the remote API. So in the unit tests we mock out the User object's "find" method to return an empty set... (ie no users were found that match this login field). But then (immediately after creation), we need to be able to find the user - we we re-stub out User.find to return the given user - otherwise the test fails.
... but when you're doing a functional test, all of the above operations are inside the atomic post :create call. You can't stub out one half, then stub out the other half of the operation partway through... because you can't get to the user object partway through the process... so creates were suddenly failing badly with seemingly no solution.

The code didn't do what I needed, and on top of this was tangled and messy and really too ugly. When it comes to rails, ugly code screams out for another solution...

Take 3: A real API

So, I turned to (what I see as) the lesser of two evils. I have created a test-API project that will simulate the real remote API with the models that are used by the (local) project. the test environment calls this API instead of the "real" one by using a defined constant for the site.

In setup I send the ActiveResource model a delete_all which will clear the remote db[1]. This simulates what Rails testing does anyway (ie clearing out the db before each test).

So now the project tests against a real, live-but-fake API that I can run in another window on my local machine. The tests run - they look pretty much the same as my ActiveRecord tests - which means I can check everything I need to check. It also means I can be assured that the code hasn't plastered over a bug with mocks and stubs.

Conclusion:

Pros:

  • Real end-end tests
  • Dynamic remote object instead of static mocks (means you can test that values are changed and that reloading returns what you expect)
  • Fewer assumptions means the test-code is less likely to be buggy - ie you're more likely to be actually testing what you think you're testing.
  • Easier to test as you don't have to mock out things that will return things that you've mocked...
  • Above leads to a more natural Railsy test syntax.

Cons:

  • Keeping two projects in synch
  • extra development time of the fake API
  • Will need to make sure the API a) exists and b) is running before you can run tests
  • Have to verify that the mock API does what the real API does.
  • Tests run slower as there's the (local) network turnaround time to consider.

In my mind, the pros outweigh the cons. YMMV

Notes:
[1] We're currently not using fixtures anyway, but it we were - I would then send the remote API a "please load up your fixtures" command.

Wednesday, 22 April 2009

should_set_the_flash better

should_set_the_flash_to doesn't do quite what I'd like. I want to: a) specify which level of flash should be set (ie make sure it's a notice and not an error) and b) not care exactly what notice has been set.

IMO tying your tests down too hard to ephemeral strings is annoying... what is somebody changed the wording from "user created" to "thanks for signing up"? You have to change your test cases for that? :P So, being able to just enter "anything" should be possible.

Note - it piggybacks on the existing flash shoulda macro, and thus allows you to use it as before (pass 'nil' for level if you want - but it's better just to use the original if you need it.

So, here's my updated code. Dump it into the bottom of your test_helper file.

class ActionController::TestCase
  # Make sure that a message has been set at the given flash level
  # you can test that a notice has been posted, but no error thus:
  # should_set_flash :notice
  # should_not_set_flash :error
  def self.should_set_the_flash(level = nil, val = :any)
    val = /.*/i if val == :any
    return should_set_the_flash_to val unless level
    if val.blank?
      should "have nothing in the #{level} flash" do
        assert flash[level].blank?, "but had: #{flash[level].inspect}"
      end
    else
      should "have something in the #{level} flash" do
        assert !flash[level].blank?, "No value set of given flash level: #{level}"
      end
      should "have #{val.inspect} in the #{level} flash" do
        assert_contains flash[level], val, ", Flash: #{flash[level].inspect}"
      end
    end
  end
  # convenience method for making sure nothing has been set for the given
  # flash level
  def self.should_not_set_the_flash(level = nil)
    should_set_the_flash level, nil
  end
end

[Edit: 22-Apr-2009] : ADDED a convenience method for "not_set" and updated "set" to take advantage of this. Also updated comments to include an example of use

Thursday, 16 April 2009

rails gotchas: nested transactions

So I've been adding some new plugins for testing that I've never tried before, in this case: shoulda and stump, and found that stump just isn't playing nice with SQLite :P

The problem only occurred when I tried to declare a proxy! function on an existing model. ok, I didn't get around to trying everything, so maybe it occurs on other things too, but it seemed to be happy with stub and mock.

The code would simply fail with the following error:

SQLite3::SQLException: cannot start a transaction within a transaction
<insert useless backtrace here>

It's been driving me crazy for the last few days trying all sorts of ways to get it to work. I almost abandoned transactional fixtures entirely before I finally found this snippet at the very *bottom* of the ActiveRecord::Transactions page in APIdock

"Most databases don’t support true nested transactions. At the time of writing, the only database that we’re aware of that supports true nested transactions, is MS-SQL."

So the problem is not transactions... but SQLite's lack of support of proper nested transactions.

The solution: Install MySQL.

It's annoying to have to go back to creating a MySQL db/user and granting rights before I can run my tests - but if it makes the test code work, then it's worth that extra step.

SQLite is nice, but it's ability to blow away the db by just doing a quick rm isn't worth not being able to properly test my code. Besides - the real db is on MySQL anyway, so I figure I might as well.

Thursday, 9 April 2009

Rails gotchas: shoulda not_allow_values_for

If you're using should_not_allow_values_for and getting a failing test something in the lines of:

Failure:
test: User should not allow email to be set to "b lah". (UserTest)
    [/usr/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.10.1/lib/shoulda/assertions.rb:56:in `assert_rejects'
     /usr/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.10.1/lib/shoulda/active_record/macros.rb:174:in `__bind_1239266378_671612'
     /usr/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.10.1/lib/shoulda/context.rb:253:in `call'
     /usr/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.10.1/lib/shoulda/context.rb:253:in `test: User should not allow email to be set to "b lah". ']:
Expected errors to include "is invalid" when email is set to "b lah", got errors: email is too short (minimum is 6 characters) ("b lah")email should look like an email address. ("b lah")

You can see in the above test that the email does have an error on it - but the error message is not the default. shoulda specifically checks for the error message - and if you don't have the default, then you need to pass it in thus:

should_not_allow_values_for :email, "b lah", :message => Authentication.bad_email_message

You shoulda test your plugin!

So, lets assume you have a plugin "acts_as_teapot" that you want to include in some of your model classes. You've written a number of useful methods for teapot-like models to use, and now want to check that the models that are implementing acts_as_teapot actually are able to make use of the full spectrum of teapotly functionality.

You could write a set of asserts that you can tell your plugin's users to "please include these asserts/tests in your model classes"... but it's not DRY, and the users might miss one, and they'll get out of date real quick... what you want is kinda one big assert they can just put in once and that calls some library on the plugin itself (so it keeps up to date with the latest plugin code).

Luckily, shoulda is here to save the day! You can create a big context full of all the right tests and save it in the plugin file itself. This will be drawn in by the model at the time it's included. Then the user just has to call a single "shoulda" and it's all done for you.

Now, I was against shoulda for a long time - mainly wondering why anybody would use a half-arsed version of rSpec if they didn't actually want rSpec... but for me, plugin-testing is the killer-app that forced me to re-evaluate shoulda, and so far it actually looks ok. :)

So, to the code...

Plugin code

module Acts
  module Teapot
    def describe_me
      "short and stout"
    end
    def tip_me_over
      "pour me out"
    end
  end
end
class Test::Unit::TestCase
  def self.should_act_as_a_teapot
    klass = model_class

    context "A #{klass.name}" do
      setup { @new_klass = klass.new }

      should "respond to teapotly functions" do
        [:tip_me_over, :describe_me].each do |f|
          assert @new_klass.respond_to?(f), "#{klass.name} should respond to the function: #{f}"
        end
      end
      should "be short and stout" do
        assert_equal "short and stout", @new_klass.describe_me, "#{klass.name} is a funny-looking teapot."
      end
      should "pour me out" do
        assert_equal "pour me out", @new_klass.tip_me_over, "#{klass.name} doesn't make very good tea!."
      end
    end
  end
end

and testing the model:

class MyTeapot < ActiveRecord::Base
  acts_as_a_teapot
  #...
end

class MyTeapotTest < ActiveSupport::TestCase
  fixtures :my_teapots

  # plugin contexts
  should_act_as_a_teapot

end

There's a real-world example in the paperclip shoulda test

rails gotchas: assert_raises a syntax error

I found that assert_raise was causing a syntax error of the form:
syntax error, unexpected '{', expecting kEND (SyntaxError)
For a fairly simple test:

should "remove it from the database" do
  assert_raise ActiveRecord::RecordNotFound { User.find(@uid)}
end

Looks like it's getting confused about nesting. the solution? parenthesise your Error thus:

should "remove it from the database" do
  assert_raise(ActiveRecord::RecordNotFound) { User.find(@uid)}
end

Monday, 6 April 2009

rails gotchas: restful_authentication not 2.0 compliant?

restful_authentication[1] is a great plugin, but the standard svn version is showing it's age. It's a whole month or so out of date... which is, of course, an eternity in the fast-paced world of Rails.

Luckily it's actually just moved home. You can find the up-to-date version on github

[1] Formerly known as acts_as_authenticated

rails gotchas: HttpMock not enough variables

A quickie for my own remembrance. I'm currently setting up HttpMock to test ActiveResource. I'd set up a few extra "routes" for it to mock out and kept getting the error below:

NoMethodError: undefined method `size' for :not_found: Symbol

For the code line:

mock.get    "/users/#{uid}.xml", {}, :not_found

The fix was pretty simple - I'd just accidentally missed out the "nil" for the body - ie it was actually an ArgumentError - but not getting picked up. ie, the code should have been:

mock.get    "/users/#{uid}.xml", {}, nil, :not_found

Thursday, 2 April 2009

ActiveRecord::Validations in ActiveResource

The holy grail for ActiveResource users is for ARes to actually behave like ActiveRecord. In theory, ARes is just like AR, but in practice it's only kinda, sorta like AR... but missing a few bits that really seem to make all the difference.

ARes is still missing fundamental functionality that we have all grown to know and love... It all looks alright on the surface, but you can't help but notice the giant glowing absence the moment you decide to hide your models away in a Web service and then try to use ARes to implement a Railsy front-end.

Needed functionality includes:

  • Associations (ie has_many/belongs_to)
  • the usual suspects of callbacks (eg before_save)
  • safe-making your attributes (eg attr_accessible)
  • Widget.count
  • Actual conditions in finders (eg :conditions => {:name => 'Joe Bloggs'}
  • and the all-important Validations

I need my validations. The funky way rails handles AR errors is one of the things that makes Rails so special. I love to be able to just type validates_presence_of :foo and for everything else to Just Work.

ARes doesn't bother with them at all - and in that case I hardly see why Rails can call it AR-like when these are missing. Oh yes, sure, you can overload the validate method on your model object, but that seems very crude! Like having to hand-write your database connection code for each model. :P

All I can say is why can't I have validates_presence_of independently to the database connection? It's not really necessary - so why are Validations still locked inside the db-wrapper?

In my opinion, ActiveResource needs a lot of upgrading. Unfortunately, that looks like a fair bit of work... and we don't know how long that will take, and whether it will just be superceded once Merb merges with Rails.

Luckily we have an interim solution, in the form of a plugin called HyperactiveResource. It is fairly crude, but it seems to roughly give us an ActiveRecord-like interface that works fairly well.

Before I hit the code, the plugin already came with a lot of the currently-missing functions - rebuilt with ARes-style processing. They also had a rough implementation of Associations (not entirely AR-like, but getting there). I've been working on adding validations and validation-callbacks.

I can't say it works perfectly as I've just started working on it, as of this morning; but I've already got the basic validations working without falling over completely and I'm using the models auto-generated by restful-authentication to test it.

It's a start...

git gotchas: the wrong forking repository

So I'm pretty new to git, really, and am learning all the places that I can stumble and fall head-over-arse. Today's escapade is entitled "how to recover from forking the wrong repository in github". It's quite a simple recovery, and you'd think it'd be obvious how to recover from it... you'd also think that github would have a "how to" in the main help page...

This solution is based on having just made the mistake and wanting to just delete it and start over straight away. Obviously this doesn't work if, say, you've started making changes and want to keep them - you're on your own there.

The solution... delete the repository and start again

  1. Go to your version of the repository on github
  2. Choose the "Admin" tab (from the list at the top)
  3. Right down the bottom of the page is a section labelled "Administration". At the bottom of the box is a small, blue link labelled "Delete this repository".
  4. Click that link - then click "Yes" a few times to convince github that you really are sure.

Now you're done. The repository may not be deleted for a few minutes (it's a background process) and your dashboard may be cached so it may appear for a little while longer. Once it's gone you can go back and pick the correct fork of the repository that you want.

Tuesday, 31 March 2009

State of authentication

We're using restful_authentication for login - because it's a good standard set of tools. However, we didn't want activation for our system, as it's just a slow-down on the path to user kick-ass-ness, and we wanted out users up and running with a minimum of fuss. However what we *did* want, was statefulness (so we could suspend users).

When I ran the usual generator without the --include-activation flag, I figured that would be the trick...

Unfortunately, AR doesn't seem to grasp that you can have statefulness without activation - so the authentication/authorisation sections have an activation state (and set up an activation code and require that you provide it and activate your user... etc) regardless of your choice of --include-activation.

This is a bit annoying - especially because the Authorization module is now also hidden deep in the bowels of the AR plugin itself (rather than in the user model), which means I can't just delete the bits I don't want.

The plugin itself is not clear about how to go about setting up your own set of states (eg removing activation, or adding a "locked" state), so this is a quickie guide.

Setting up your own states

To begin with, to override your states, you need to copy the appropriate Authorization file into your lib directory eg:
cp vendor/plugins/restful_authentication/lib/authorization/stateful_roles.rb lib/my_stateful_roles.rb

Then require this file at the top of your user model eg:
require 'lib/my_stateful_roles.rb'

Open up this file and change the module name so you don't clash (in the bizarre case you need to use the standard set of states in some other model) eg:

module Authorization
  module MyStatefulRoles

Your user model will then have a reference to this module that you need to update from Authorization::StatefulRoles to Authorization::MyStatefulRoles

You now have a file that you can alter to your heart's content.

Coming soon: how to successfully remove the activation state from this file...

Monday, 30 March 2009

acts_as_state_machine 2.0

I spun up a new rails project and downloaded my favourite authentication plugin restful_authentication, only to find that all out-of-the-box tests break (oh noez!)

Clearly, I thought, restful_authentication is now just a millisecond behind edge rails... and therefore doesn't work properly anymore.

I quickly realised that no, it's just that the old-standard install (via svn) is no longer maintained - you have to move over to the github version.

This solves most of the problems, but I was still having a problem with one of the tests. It seems that acts_as_state_machine is similarly non-Rails 2.0 compliant.

Luckily, acts_as_state_machine has just a one-line fix to get it back up to scratch. It seems that the write_attribute method has been deprecated, and this causes a little bit of internal havoc for the state? style of methods (eg passive?, active? etc as used by restful_authentication).

The test cases kept giving me:

NoMethodError: You have a nil object when you didn't expect it!
The error occurred while evaluating nil.to_sym

To fix it, search for the string: "def set_initial_state", then replace the line (just below it):
write_attribute self.class.state_column, self.class.initial_state.to_s
with:
self.send "#{self.class.state_column}=", self.class.initial_state.to_s

I just want a toaster...

I was buying all my basic housey stuff at Tesco today (it's *that* kind of supermarket/superstore), but feeling overwhelmed... Being in a foreign country means all the brands mean nothing to you, so you don't know which ones do what, and you find there are many brands that do things you've never even heard of.

So you have two options. You can either pick things at random and hope you aren't going to end up with your washing machine spilling a river of frothy soap suds across the floor, or check every single item to make sure it does what you expect it to do.

When you have to read the back of every bottle of washing liquid and read all the dodgy manufacturers claims that mean nothing to you to decide whether you want "unscented" or "hypoallergenic" or "automatic" or "non-biological"... and whether that actually means anything or just another manufacturer's claim (like "full of wholesome goodness")... for every single item in the shopping trolley you find it's quite easy to take several hours just to do a normal shop.

By the end of that you come to a shining wall of toasters and find yourself weighing the benefits of "hi-lift" vs "variable-browning" and whether it's better to have "sneak peek" or "cool wall" technology and you get to the stage where you just want to scream "I want something that takes bread and spits out toast, that's *all* I want!"... :P

...and since when did toasters all become so *huge*. I'm missing the toaster I had in Sydney even despite having a dodgy Singapore plug that required a plug-adapter to use. It took four slices of toast and fit into nearly the same space as my new two-slice toaster that's tall and fat and looks like some kind of shiny metal alien pod. :P

and in case you ask "well why didn't I choose one that doesn't look like that - they all do, of course. Apart from the ones that are even *bigger* and look like some kind of industrial machine that will take your bread and throw it clear across the factory floor to the industrial jam-spreader...

...in the end, of course, I went with the one that had all of the above. Because I can't be sure I won't suddenly need "cool wall" technology (whatever that is)... I can always google it when I get home.

Saturday, 7 February 2009

Some day, my visa will come...

So in case you happen not to have actually spoken to me any time in the past two months, I'm in the process of moving to the UK. Of which, the major step is to get myself a UK work visa.

I'm (frankly) too old for a holiday working visa (and I don't want to go for that short a time anyway), so what I need is the Tier 1 general migrant visa. This will let me work in the UK for any company for up to 3 years before having to reapply, so it's pretty good.

However... apart from some very strict requirements the current wait for a visa of this kind is a bit on the long side.

The website tells us that there is a wait of 25 working days for any kind of visa[1], but this is simply not the case. The wait is currently averaging 45 working days. The following thread is an ongoing conversation by a large number of people frustrated about the current Tier 1 visa waiting times. It's been running constantly since October (when the Tier 1 was first introduced) and now runs at 47 pages of comments... this should give an indication of the current level of feeling surrounding the handling of this class of visa.

One of the chief grumbles is that the official website flat-out lies about the length of time it takes. The visa-office staff are aware of how long the wait times are currently averaging (I called and asked).

Moving internationally is a really large, life-changing event. It takes a lot of planning. It requires packing up your house and quitting your job, and long, tearful farewell parties... If we are told that it takes 25 working days, and then it takes another month... it means there are a lot of people left twiddling their thumbs and watching their bank balance go down - at an alarming rate if we're unlucky enough to have quit our job or are forced to stay in a motel after giving notice or selling the house.

Don't get me wrong - I don't mind that it takes 45 working days, but if I had known this from the start, I wouldn't have planned my life around 25 working days... and things would be going much smoother. As it is, I've been interviewing with several companies in the UK and having to tell them "looks like my visa *still* hasn't come through yet... can you hold off another 2 weeks"... which is a situation that *nobody* likes to be in.

It's not my fault that the visa simply hasn't come through, and that I have no idea of if or when it ever will... but I can't shake the feeling that I'm coming across as being flaky... which is a feeling that I really dislike.

The problem is compounded by the fact that there is literally no way in which one can find out the progress of a visa application[2]. There is no website, there is nobody I can call. I just have to wait and hope... and wait and hope some more... while every friend, relative and innocent bystander is going crazy alongside me and asking me when it's going to come... and I don't have any answer to give them!

Being my mother's daughter, I called anyway and asked - they said there was no way, but they confirmed it was taking 45 working days, so they know that the website is incorrect, and yet have done nothing about it.

That made me mad. Are they deliberately misleading people or are they simply too negligent to bother? We're not talking about the kiddies on their gap year holiday doing a bit of waitressing on the side (incidentally, those visa come through in a jiffy). These are the most highly skilled people in the country who are moving their lives around to get over there, and the official website gives an estimate that is 80% inaccurate... and they *know* it is! :(

The real kicker for me is that, because we can't see our progress, they have no obligation to get anything done at all! Now, I know the system has just changed and they probably have a stack of applications a mile high on their desk - so they're probably *not* just being lazy... but the lack of transparency certainly leaves the system open to abuse.

One final niggling point is that they recently published some world-wide stats for visa processing times... and it looks like countries such as the US have an average turnaround time of only 6 working days. I can imagine a time difference of maybe ten or even twenty days, but this is ludicrous. What could cause a discrepancy of over 600%!

If there's a shortage of staff... hire more?

[1] Apart from a permanent settlement visa, or an annoyingly vague description of a "non straight forward" visa - which apparently means a visa application that is incomplete... by which assumption the Tier one should be straight forward as long as you provide everything they ask for.
[2] at least until after 45 working days have past in which case you can pay $10 to get an SMS "progress report" sent to you (of which the aforementioned thread has several stories of it never arriving) and which, when it arrives, tends to say something vague along the lines of "In progress", and with no indication of how long it will take from there.

Wednesday, 31 December 2008

Link: A signature cadence

Too many websites (and applications) are soul-less, corporate entities that treat you like just another number.

Rands' article: A signature cadence shows how a human tone can make a big difference in how we view a site. Even tiny details like the copyright wording can help make an overall impression that you are actually a bunch of real people. I think we need more of that.

Tuesday, 30 December 2008

Squidoo and ssh tunneling

I've found squidoo - a site where you can set up pages called lenses that help direct people to all the relevant information about a specific topic.

I love it!

I've only just begun, but I think I'll slowly add pages on my favourite topics and see where it goes.

ssh tunnels R Us

To start with, I put up a page on ssh tunneling.

For years I've used ssh to login to a remote linux box to get work done. But recently I had to get through to the http server on my work machine - which was hidden behind a firewall. The best way to do that, it seems, is to set up an ssh tunnel and just port-forward the http port to my local machine!

In the process I found several useful (and less useful) articles on the subject, and it took me ages to crawl through each one to figure out which was which. It seems "ssh tunneling" means different things to different people...

So where does squidoo feature in all this?

In the squidoo lens, I gave an overview of ssh and tunneling - and then listed each of the articles with a brief description of the contents. In this way, the squidoo page acts as a differentiator for any people searching for the same thing in future. Hopefully it'll cut down the time people need to spend scanning all the possible google results they can find. They can zero in on an answer all the more quickly.

[Edit...] More squidoo...

So I've now gone and added a few more pages on my other hobbies.

For starters, I quite like brewing mead - which is a wine made from honey. I've been doing it for about 15 years now, and you can read more about it in my lens on mead, or you can jump straight to my (long) howto lens that will teach you how to make mead in three weekends.

Alternatively you can try your hand at home preserving with vacola - an ingenious system of bottling that creates a vacuum seal in the bottle to hold on the lid.

Stay tuned for more fun stuff as I go along.

Thursday, 25 December 2008

Link: 13 coding nightmares to avoid

Here is a great list of bad RoR practices - make sure you read the comments as there are some interesting exceptions raised.

Merry Christmas!

So I've been pretty quiet for the past month. That's because I'm running around like a mad thing getting set to make a big move to the UK. Won't be for another month yet, but it's consuming all my spare time.

Anyway, today's been quiet enough to sit down and actually write something so - merry christmas everybody!

Saturday, 29 November 2008

I wrote a novel - w00t!

Woo hoo - much celebrating as I have successfully finished 50,000 words for NaNoWriMo! You can see my novel info here - though not the novel itself, that's still only a first draft!

Now to party for a while. :)

Wednesday, 5 November 2008

KioqKioqKioq OR fixing mutt's base64 problem

So I've occasionally been getting *really* useful email messages such as:

KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KgpUSElTIElTIEFOIEFVVE9NQVRFRCBFTUFJTCAtIFBMRUFTRSBETyBOT1Qg
UkVQTFkuCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq
KioqKioqKioKCgpEZWFyIE5ldyBNb25leWJvb2tlcnMgQ3VzdG9tZXIsCgpZ
b3UgaGF2ZSBjaG9zZW4gbWJfdGVzdF9jb21wYW55QHRhcnluZWFzdC5jb20g
YXMgdGhlIGVtYWlsIGFkZHJlc3MgZm9yIHlvdXIgbmV3IE1vbmV5Ym9va2Vy
cyBhY2NvdW50LiAKClBsZWFzZSB2ZXJpZnkgdGhpcyBlbWFpbCBhZGRyZXNz

Not having the slightest clue as to what was causing it - I'd simply return them to the sender and ask for help... sadly the senders generally didn't know what it was either.

It turns out that the sender's setup is incorrectly labelling these emails as 8bit... when it's actually been encoded in base64. To fix it:

  1. Save the email to its own new folder (just so there's only one email in the file to edit).
  2. Open it up in a plain text editor (eg vim)
  3. Update the header line that reads:
    Content-Transfer-Encoding: 8bit to
    Content-Transfer-Encoding: base64 - save and close
  4. Navigate to the folder in mutt and read away happily.

Sunday, 2 November 2008

National Novel Writing Month

It's November - and that means it's time to whip out a pen and start a frenzy of writing in the hopes of finishing up a new masterpiece... or at least 50K of whatever comes to mind during this month of madness.

It's National Novel Writing Month (or NaNoWriMo for short). The site was slashdotted for the first day and a half, but the servers seem to be taking the load now - which means we have a chance to actually update our wordcounts and read the forums.

NaNoWriMo is a personal challenge to start writing a novel on November 1st and aim to have 50,000 words by midnight on November 30th. Last year there were over 100,000 participants - of which 15,000 passed the 50K mark by month's-end. I was one of those 15K :)

Here's a link to my author page so you can see how much I've done so far this year. I'll also add a wordcount widget onto the blog for the duration of the month (should be there in the left-hand column).

It's heaps of fun and it's open to anyone. So, if you're up to a unique kind of personal challenge, join us in our month of writing. Or just come along and see what all the fuss is about!

Wednesday, 29 October 2008

Rails a little over-protective?

Was busy developing and minding my own business, when suddenly my functional tests all stopped working giving me this error:

ActionController::InvalidAuthenticityToken: No :secret given to the #protect_from_forgery call. Set that or use a session store capable of generating its own keys (Cookie Session Store).

Now we happen to have just switched on the use of the cookie store - and the actions all work just fine from the browser. Checking the forms even shows up the authenticity tokens etc... this only occurs during testing.

Googling gives a few rspec-based solutions, but very few are even possible for Test::Unit. I tried disabling config.action_controller.allow_forgery_protection - but that did a big nothing for me. So I bodgied up a quick-n-dirty workaround just for the test environment as per below.

Right now I still don't know what's causing the bug, but this gets me past it until I can figure out what's actually wrong. YMMV :)

  # See ActionController::RequestForgeryProtection for details
  if RAILS_ENV == 'test'
    # dodgy workaround for Rails 2.0 bug in functional tests - which don't
    # seem to use a cookie store properly. Reference to issue:
    # http://groups.google.com/group/railsspace/browse_thread/thread/fcdbfa4e65bf86de
    protect_from_forgery :secret => 'c1c6ebaee01fecc9aa9bc105d235b2c2'
  else
    # Uncomment the :secret if you're not using the cookie session store
    protect_from_forgery # :secret => 'c1c6ebaee01fecc9aa9bc105d235b2c2'
  end

Tuesday, 28 October 2008

Headhunters strike out again

A follow-on from my negative experience with headhunters. I've begun receiving exactly the sort of "job offers" that I was fearing. I've got offers (from all over the world, mind) asking me if I'd like to work as (for example) a junior developer on an embedded-kernel contract in Belgium... which'd be nice and useful if I were interested in that sort of thing instead of having only 6 months non-commercial experience (at uni) doing this stuff as a elective subject... or at least held a work-permit that allowed me to work in Belgium.

The end result makes it fairly obvious that their recruiters are just too damn lazy to do more than a quick search on the db... followed by spamming all the results. They've ripped out the db-friendly data (ie my naked skillset) and divorced it from the important stuff (ie what sort of job I'm actually interested in). Leaving me to be spammed by every idiot with a contract to push.

If I were a junior developer, desperate for any job I could get, I'd probably just have to suck it up... but I'm not - and they're alienating an in-demand developer with lots of experience. This doesn't make me feel like responding positively to any of their future offers... even if they do suit my requirements.

What gets me is that it'd be so easy to make this system work. Even if they had a few extra fields in their db for "what type of job is the candidate looking for", they could run these as a filter over their resultset and turn that spam into much-less-despised targetted advertising.

I'd even be interested (possibly even grateful) to receive such a service - instead of so annoyed that I've blogged about it twice.

Enumerable.average

Was looking around for a quickie function to do an average on an Enumerable/Array just like max and min - and found there isn't one. So here's a quick-n-dirty one that you can drop into config/environment.rb. Examples for use in the comments:

module Enumerable
  # returns a simple average of each item
  # If a block is passed - it will try to perform the operation on the item.
  # if the result is nil - it won't be counted towards the average.
  # Egs: 
  # [1,2,3].average == 2
  # ["a","ab","abc"].average &:length == 2
  # @purchases.average &:item_price == 10.45
  def average(&block)
    the_sum = 0
    total_count = 0
    self.each do |item|
      # either the actual item, or a method called on the item
      next_value = block_given? ? yield(item) : item
      unless next_value.nil?
        the_sum += next_value
        total_count += 1
      end
    end
    return nil unless total_count > 0
    the_sum / total_count
  end
end

I used it for displaying neat averages in the view thus:

    <p>Averages:
    <br />Number of items: <%= @purchases.average &:num_items -%>
    <br />Unit price: <%= @purchases.average &:price -%>
    <br />Total price: <%= @purchases.average {|p| p.num_items * p.price } -%> </p>

Thursday, 23 October 2008

That doesn't help me, it helps you

I just got off the phone with Australia Post and I'm not happy. Even though they were very polite to me, they failed to provide a good customer service experience. This issue requires a bit of backstory.

While in Melbourne last weekend, I'd bought two half-cases of wine and paid for them to be delivered. I have no car, so I left explicit instructions for the parcels to be left on the back doorstep if I was not at home.

I got one of the little blue cards in the post yesterday, telling me that they tried to deliver one of the parcels to me in the 1 hour when I was out of the house getting some lunch. I figured that the winery must have forgotten to put the "in case not at home" message on the parcel, so today I decided to just fetch it from the post office.

I damaged my knees fencing, a couple of years ago, so walking eight blocks while carrying a heavy box is painful for me. My arms and knees both ache, now, but I got the parcel home... only to find a second little blue card in the mailbox.

The parcel from yesterday clearly has the "in case not at home" instructions printed on the outside, so I figured I'd call up Australia Post and see if they could actually do what I paid them to - which is to deliver my second parcel to my house.

I called them up and waded through their long telephone menu-system to get to a real human being. The man at the other end seemed kind enough, but the only option available to him was to allow me to lodge a complaint against the contractor who failed to follow the given instructions.

I don't really want to lodge a complaint - I just wanted my parcel delivered - which is what I paid for.

I asked the man if he could find anyone that would be able to help me get what I wanted. He went back to his supervisor, leaving me on hold, and returned to tell me that the only way I could get it redelivered was to call up the people that I originally bought it from (some winery out in the Yarra valley - I don't know which one as the parcel is still sitting in the aussie post office), and to get *them* to call aussie post and figure out if their contract supports "redelivery-on-failure"...

Good grief! From my POV it's pretty simple:

  1. I paid for delivery to my home
  2. Aussie Post stuffed it up
  3. Aussie Post should find a way to get it right

Instead of finding a way to get my parcel to me, they kept insisting that they were helping me by starting up a complaints process and making sure it "never happens again". If you want to reprimand your contractor, that's your business. It will help you improve your own processes over time, and no doubt his non-instruction-following is the weak link that broke in this particular chain of events...

But doing this isn't going to help me. Getting my parcel to me helps me - that's all I want, and Aussie Post wasn't willing ot help in that regard. I was a customer on the line right then and there with a problem that Australia Post caused - and wasn't able to get any resolution that actually solved that problem.

I was left with the feeling that Australia Post wasn't willing take responsibility for this mistake and do something about it. Instead they wanted to shift the blame to the contractor, and then claim that responsibility was out of their hands.

This reeks of bad customer service.

If you want to leave a good impression with your customers, follow one of the basic rules of human conduct: if you stuff up, take responsibility and try to fix it. Whatever you do, don't just give a reassurance that it won't happen again... we won't believe you. After all, you've just proven that you can't be trusted. At least if you try to find a way to give me what you promised in the first place - it'll go a long way towards reassuring me that you can act in good faith.

Tuesday, 14 October 2008

soap4r + rails can't find SOAP::Mapping::EncodedRegistry

Joining a new project, I had to go through the usual setting-up rigmarole - which means finding several new things to break/configure/reinstall. One of which was to get soap4r to run.

I'd run gem install soap4r just fine, but running rake was giving me the error below:

rake aborted!
uninitialized constant SOAP::Mapping::EncodedRegistry

Scrounging around gave several solutions that didn't work, mainly using two of the three lines below. I required all three of the following before it ran for me.

# Add these lines to config/environment.rb just before the Rails initializer code
require 'rubygems'
gem 'soap4r'
require 'soap/mapping'

Tuesday, 7 October 2008

Words and pictures

Ok, this is a neat toy: Wordle is a site that runs a java applet that takes some text (or a URL for a feed) and make a pretty word-picture... like the one below for this blog.

Hmmm... I wonder if I can use it to improve the editing of my blogposts. Looks like I use "going" far too much!;)

Sunday, 28 September 2008

Headhunters and a dearth of trust

What is it with some headhunters?
I had a phonecall from a headhunter the other day that really rubbed my fur up the wrong way.

First, they wanted my CV in Word format. I hate that. Quite apart from having to explain that I use Linux - which frequently leads in to the "what is an OS" discussion that I find so annoying coming from an IT recruiter. Yes I can use OpenOffice to create a Word doc, but it's a preference thing. I prefer to write my CV in PDF so I have more control over the formatting - because I've found that .doc format "helps" too much, and suffers from bit-rot.

But it's not like they can't read/print-out a copy of my CV if it's in PDF. What really gets me is that there are only two reasons for wanting my CV in Word. First is so they can copy/paste my skills/experience into their mammoth database - reducing me to a set of numbers... which I really don't like.

I recognise that might be a knock to their usual system, but I really don't like being just a number in an uncharted sea of applicants. It doesn't work for me to be recruited by a company that would view me as only so important as the number (and not the quality) of the years of experience I have.

Second is that they want to remove my contact information so that they are the only point of contact between me and the employer. This view is especially reinforced by headhunters that refuse to give you the prospective employer's name, industry, size or any other potentially-identifying details (or politely change subject when I ask, as this one did). How the hell am I supposed to decide if I'm interested in working for a company I know nothing about?

I should point out - I get a phonecall a week from various recruiters. I don't have time to go to an interview a week - so I have to decide, based on what you tell me, if you are worth a morning of my time.

Word to the wise: recruiters perform a valuable service - both to employers and employees. If you are good at what you do, then nobody will be unhappy about paying you for your services! The only recruiters that need to resort to cheap tactics like hiding names are the ones that cannot build a reputation for excellence on their own... thus if you flat-out refuse, then it instantly gets my suspicions up.

Yes, I'm sure you will get ocasionally burnt by employees and employers that are unscrupulous and try to screw you out of your fee. But are those the kind of clients you want to keep anyway?

Recruiting is a people business - it is outsourced HR and builds upon long-term relationships. Trust is important, and screwing around like this ruffles that layer of trust and just feels tacky.

Please don't do it.

Thursday, 25 September 2008

Rspec mocks too much

I've recently had to learn the other side of testing - ie rspec, due to my new contract having a test suite written in "the other testing suite" * ;)

Seems fairly straightforward so far, but there's one big worry I have: Rspec has a huge reliance upon mocks.

Now I completely understand (and endorse) the benefits of using mocks. It means you can independantly test a model/controller/whatever without it mattering if something happens to have broken in some other model/module/whatever. This is a Good Thing and allows for more accurate testing due to all the benefits of loose coupling. However, I get a bit worried if everything is mocked out and there are no integration-style tests at all.

For example, if I'm testing a view, and using rspec best practice I've mocked out all model objects and don't bother with the real thing. What happens if a model changes (say a column is removed that shows up on an index list)? The view specs all rely on a mock that explicitly states what is returned, so the view specs all still pass. But the index page will be broken for a user that actually uses the site, because the real object doesn't bahave that way anymore. The mocks and the model are out of synch. :P

In an infrequently-used area of the site, this sort of error may go undetected and slip through to release.

This worries me. I know I like to rely on my test suite to catch things like this wherever possible. Sure I generally do a manual click-through, but I know that this is unreliable especially WRT edge-cases.

So, am I needlessly worrying? Have I simply missed something about how rspec is supposed to be used? What's best practice in this situation?

*Which admittedly feels a lot like having to switch from vi to emacs... Sure I know that "the other one" works for lots of people, and I meant to get around to learning it someday...

Tuesday, 23 September 2008

Getting ffmpeg to work under ubuntu

I had a fair whack of trouble getting ffmpeg to compile on ubuntu. These are the steps I took that finally got it working for me.

Firstly - I'm going to assume you've downloaded and installed the appropriate libraries (codecs?) for the various audio/video formats that you are going to use (eg lame). I'm just going to cover making ffmpeg actually compile and run.

The usual set of steps will be

  • Downloading ffmpeg: svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
  • cd ffmpeg
  • Configuring appropriately to your build. Eg: ./configure --enable-libmp3lame --disable-vhook --disable-mmx --enable-nonfree --enable-gpl --enable-shared --enable-libamr-nb --enable-libfaad --enable-libx264 --enable-encoder=x264 --prefix=/opt/local --extra-ldflags=-L/opt/local/lib --extra-cflags=-I/opt/local/include
  • make
  • sudo make install

If that all works for you, then brilliant, but for me it fell over while trying to make, telling me it couldn't find x264. For some reason it's necessary to add --enable-pthreads to make it work on ubuntu.

The next big hurdle for me was that a bug appears in the v4l2 library for ubuntu. It doesn’t properly load some of the basic linux types. This is apparently a well-known bug in some of the older linux headers, and will stall the compilation of ffmpeg with a whole slew of bogus error messages, that essentially boil down to the __u64 and __s64 types failing to be defined appropriately.

To make a temporary fix, you can open up asm/types.h (I used find to find it) and figure out where the asm types are located on your distro (for me they’re in asm-i386/types.h).

Now, copy the two lines that define __u64 and __s64 and paste them into:
/usr/include/linux/videodev2.h just above their first use to help define v4l2_std_id

Note - this is an ugly, quick hack, but it should get the compilation going again. For a more permanent solution - I'd suggest chatting to the ubuntu people... or maybe it's time for me to get around to upgrading from feisty ;)

That will generally get you compiled and installed, but I then fell over a run-time bug that boiled down to an error message: “cannot find libavdevice.so.52”. This problem had me stalled for weeks. Google returns a lot of talk about the problem, but very few solutions, but eventually I stumbled across the answer: you need to add a library path to your environment thus:
export LD_LIBRARY_PATH=/usr/local/lib/

Finally ffmpeg safely up and running on ubuntu. Yay!

Saturday, 20 September 2008

Snippet: MyModel.options_for_select

How many times have you written a quickie drop-down box to choose one instance of an associated model? Here's a quickie monkey-patch that will help. Put the following in environment.rb (or similar):

class ActiveRecord::Base
  # convenience method for selection boxes of this model
  # Assumes existance of a function "display_name" that will be a
  # displayable name for the select box.
  def self.options_for_select
    self.find(:all).map {|p| [p.display_name, p.id] }
  end
end

Make sure you have a function in your model named "display_name" that returns what you'd like to see in your drop-down, eg:

class Person < ActiveRecord::Base
  has_many :posts

  # convenience method for the person's name
  def display_name
    [first_name, last_name].compact.join(', ')
  end
end
class Post < ActiveRecord::Base
  belongs_to :person

  alias :display_name :title
end

Then use in a view (eg as follows):

<% form_for(@person) do |f| %>
<p>Pick a post: <%= f.select :post_id, Post.options_for_select -%><%= f.submit 'Go!' -%></p>
<% end %>

Thursday, 4 September 2008

Snippet: monkey-patching a gem

Need to make a quick monkey patch to a gem you're using? Follow the steps below:

  1. If you don't have a vendor gems directory: svn mkdir vendor/gems
  2. cd vendor/gems
  3. Unpack the gem into your vendor/gems directory with
    unpack gem <the_gem_name>
  4. Checkin the above changes so you have a clean version of the gem to start with
  5. Make your monkey patches and check them in
  6. Make sure vendor/gems is in the loadpath for gems

And you're done

Tuesday, 19 August 2008

Link: Why I don't have a startup yet

Great post called Why I don't have a startup yet. Snippet:

"I love what I do at Yahoo, and I care enough about what we create that I want to focus all my energy on creating value for users. It’s good practice. But as long as I work at this job, I won’t have enough left over at the end of the day to seriously invest in anything else."

But further to that - there's so much in the world to do. Doing work is part of that, but if it's the only thing in your life, then "All work and no play makes Taryn a dull girl". Sure, some coding is play - and I do love to code my own stuff too... but I like so many things, that if I focus my off-hours on coding, I'd lose sight of all the rest that life has to offer.

So sure, I choose to spend some time coding, but also a lot of time reading books while sipping great coffee, getting together with interesting bunches of people, travelling to far-flung places, writing up my travels or even writing fiction (however badly), taking photographs, attending medieval feasts and wars, making home-brew mead, or even just walking outside and enjoying the sun.

Life is an adventure... make the most of it.

Wednesday, 13 August 2008

Tact filters for geeks

I just stumbled into this great article. It described the difference between how geeks vs "normal" people filter their social discourse for tact. It makes a lot of sense, and explains my frustration with non-geeks not "talking straight"... as well as, no doubt, the shocked reactions of these same people when I do. Interesting reading.

Tuesday, 12 August 2008

Updating gem to 1.2.0 on Ubuntu

Like several others, I tried to gem update today and failed. It gave me weird error telling me it couldn't find the update gem in the repository:

%> sudo gem update --system
Updating RubyGems
Bulk updating Gem source index for: http://gems.rubyforge.org/
Updating rubygems-update
ERROR:  While executing gem ... (Gem::GemNotFoundException)
    could not find rubygems-update locally or in a repository

Googling gave me a fair few other links to people with the same issue. The main one seems to be listed under this ruby-forum topic which started as the release-notes for the Gem update to version 1.2.0. People that had problems updating to this gem listed their steps-to-reproduce in the replies below, and I tried all the suggestions.

sudo gem update --system was the original command I tried and it gave me the error as above. All the fora on the topic insisted that the workaround is to run: gem install rubygems-update -v 1.1.1, but this just gave the same error. I did note that most of these people were on various flavours of Mac, and the pleas for help from those on Ubuntu were drifting away into the sky unheard...

One forum user said he tried the above from his gem directory and that worked for him...
but not for me...

I even tried reinstalling ruby/rubygems via apt-get, but the former was latest version, and the latter doesn't exist...

I was about ready to nuke the site from orbit... and reinstall from scratch and went to the rubyforge page for rubygems. This lists the source code, but I also noticed it had individual "upgrade" gems available for download. This saved me from my previous scorched-earth plan, and eventually resulted in success. For future reference:

  1. copy rubygems-update-1.1.0.gem to a local directory
  2. then run: sudo gem install rubygems-update-1.1.0.gem
  3. repeat with rubygems-update-1.2.0.gem
  4. then run update_rubygems

Now it's all fine and sudo gem update --system happily fetches my new gems as before.

Thursday, 7 August 2008

Explaining anti-virus suckage to the tech-averse

Being the most IT-savvy member of my extended family, I'm often called upon for random tidbits of IT information. This is regardless of whether or not I'm actually likely to know anything about the subject at hand... because obviously I'm "into IT" so I should know how to network an ancient dot-matrix printer to an ageing pentium... (clue: it's probably not working for you because it's time to replace the hardware).

The one I dread most, however, is "why is my internet so slow"... the obvious candidate being "have you considered the possibility that you might have a virus?" - which is usually met with the same sort of scandalised reaction reserved for asking somebody whether they have a venereal disease.

Surely computer viruses are something that only happens to *other* people? After all, we're good, clean types that don't take part in *that* sort of behaviour... why would we have something so dirty as a virus? besides... which we always use a condom anti-virus software... so there's no possible way we could have anything right?

Depending on the tech-savviness of the audience I'll then launch into an explanation as to why anti-virus software doesn't always work... This generally goes down like a lead balloon. With the usual response being "but I have the latest <insert expensive AV software name here> and keep the signatures fresh up-to-the-minute!"

The problem being that I know this subject only enough to know that even with the latest virus signatures, it's not perfect. I don't know it well enough to really describe in laymans' terms *why*.

Today I came across an article on Coding Horror about how blacklists don't work. but it also neatly segue's into the reason why anti-virus software is always going to be one step behind the competition. I think it summarises the issue nicely - far better than I have ever been able to describe.

I know I could never give this to my grandma, but I think it's a step in the right direction - and I can probably point my partially-tech-savvy aunts and uncles in that direction from now on.

Tuesday, 29 July 2008

icon_button_to_remote

Following from my earlier post on making icon buttons, I needed to add AJAXy goodness to the edit actions on one of our index pages.... but I still wanted cute little icons to make them look pretty.

So I've updated the code to incorporate an icon_button_to_remote method. Note that icon_button_to has also been modified, but it still depends upon the get_icon_path and fixed_button_to methods from the previous post.

  # generates appropriate image-buttons, given an "icon type" and the usual
  # button options.
  def icon_button_to(icon_type, icon_label, form_opts = nil, button_opts = {})
    unless form_opts.blank?
      button_opts.merge!(:type => 'image', :src => get_icon_path(icon_type), 
                       :alt => icon_label, :title => icon_label)
      the_icon = fixed_button_to(icon_label, form_opts, button_opts)
    else
      # no form was passed in - we want a non-linked icon.
      the_icon = image_tag(get_icon_path(icon_type), :width => 16, :height => '16', 
                       :alt => icon_label, :title => icon_label)
    end
    content_tag 'div', the_icon, :class => 'icon_button'
  end
  # add a remote AJAX-linked icon button
  def icon_button_to_remote(icon_type, icon_label, remote_options = {}, button_options = {})
    link_to_remote(icon_button_to(icon_type, icon_label, nil, button_options), remote_options)
  end

The example would be for an index page with a list of Widgets. Each one has an "edit" button using the code below. The page also has a section for entering a new widget - which would be replaced by the "edit this widget" template when you click on the icon.

<!-- In the template at the top of the list -->
<%= w_path = new_widget_path() # url_for is slow
    icon_button_to_remote(
         :new, 'Add new widget', 
         :update => 'new_widget_div',
         :url => w_path,
         :method => :get,
         :html => {:href => w_path},
         :complete => "Element.hide('edit_widget_div'); 
                          Element.show('new_widget_div'); 
                          new Effect.Highlight('new_widget_div');"
        ) -%>

<!-- and next to each widget -->
<%= ew_path = edit_widget_path(@widget) # url_for is slow
    icon_button_to_remote(
         :edit, 'Edit this widget', 
         :update => 'edit_widget_div',
         :url => ew_path,
         :method => :get,
         :html => {:href => ew_path},
         :complete => "Element.hide('new_widget_div'); 
                          Element.show('edit_widget_div'); 
                          new Effect.Highlight('edit_widget_div');"
        ) -%>

<!-- Later in the template... -->
<div id="new_widget_div">New widget form would go here</div>

<div id="edit_widget_div" style="display:none;">Edit widget form will be populated here</div>

You also have to update your Controller code for the new and edit actions to accept js and respond with no layout - otherwise your div gets populated with the entire edit/new page :P

The best way is to move the meat of the page into a partial (which can also be loaded in the edit.rhtml template) callable from the action thus:

  def edit
    respond_to do |format|
      format.js { return render(:partial => 'edit', :layout => false) }
      format.html # edit.rhtml
    end
  end

Saturday, 26 July 2008

Best vs Good enough

I can't remember which I've heard - I think it was "good enough is the enemy of best", but it could as easily be the other way around. It's a bit confusing as I've definitely heard conflicting stories.

We are the best!

From one perspective, you want your new startup to be the best - because, in a lot of ways, that is what will distinguish you from your competitors. The difference in market-share between Google and Yahoo! is an obvious example. So is the difference between Amazon and... whoever comes second after Amazon.

Clearly it pays big to be the best in your niche. You need to be the best. "Good enough" not only won't cut it, but will relegate you as indistinguishable from the swamp of mediocrity in which all the other companies float.

In this sense, "good enough" is the enemy of "best".

Good enough. Ship it!

And yet, you need to *ship*, because if you don't have anything to ship, then the customers will never hear of you. This follows from the principle of "ship early, ship often". It's all too easy to get caught in analysis paralysis, endlessly perfecting what you've got without ever putting it "out there" for real users to get their hands on.

In that respect, "best" is the enemy of "good enough".

So which is it?

As with everything, you need a strategic balance.

You don't need to be best at *everything*. You'll wear yourself out trying. You just need to be best at what you've chosen for your market differentiator. Perhaps you're aiming to provide the best features for power users, or most comprehensible features for beginners, or even the nicest customer service. Whatever you decide on, use that focus to drive your quest for perfection. You can push for the best in this one area and let other areas be "good enough".

If you try to swallow the elephant whole, you'll never get it down. Instead, keep focussed on your core competencies, and don't worry *too* much about perfection in the other areas. You can always make them better in your 2.0 release.

Tuesday, 22 July 2008

More thoughts on Rails-doc 2.0

I'd been re-contemplating the Rails-doc 2.0 site and had begun to write a post about the minor annoyances I found, when suddenly up popped a new post by Jeremy on rubycorner.

I guess I'm not the only person whose experience was akin to: "OMG wow! *click* *click* hang on..."

My annoyances from rails-doc 2.0

There's not a lot wrong with rails-doc 2.0, the following are merely annoying.

Click depth

One of the benefits of the previous rails-doc site is that it's all there at your fingertips all the time. Rails has a deep hierarchy and now the only way into it is to search randomly. This can be fun - the auto-prompter is pretty good... as long as you know exactly what you're looking for. But on the old site, if you can't quite recall what you're looking for, you can browse down to the list alphabetically until you find it. With the new site you have to delete your old search and continue to try other names. Search helps, but having a browse-alternative is also helpful.

Another problem with deep click-hierarchies is that you can't get as much information at once. On the original site, all the methods are on the same page - and all you need do is click the anchor or scroll down to them to get a feel for the whole class. You don't have to keep clicking through and back if you want to look at each of the methods.

Searching code

...requires code-sensitive search. If I type "default_error_messages" it means something entirely different to a search with the words "default error messages" somewhere in the body.

version consistency

If I chose version 1.2.6 for the ActiveRecord class, chances are pretty strong that I'll want to keep looking at version 1.2.6 when I click through to the methods... or anywhere else, for that matter. I think there's a strong case for persisting the chosen version.

Is there really a problem?

As you might notice, these are just GUI annoyances not real issues, as such. So is there really a problem?

Jeremy points out that user-added notes are all well and good, but what we need is for the actual real rails-doc to be updated. That isn't going to happen here (at least with the current incarnation). His proposed solution sounds great - and I'll look forward to its release. Perhaps he should team up with the rails-doc.org folks ;)

But has it got anything right?

Jeremy actually stated this succinctly:

"If I’d known that all we needed was a good looking RDoc app, I could’ve fixed the problem a while ago."

Lets face it, we're geeks and we love shiny new toys... Rails being the epitomy of the shiny new thing. So yes, rails-doc 2.0 got one thing absolutely right - it provides a pretty interface that is easy and fun to explore. That's often all it takes folks like us to start contributing - so they got that right.

Monday, 21 July 2008

Rails doc 2.0

Finally some decent documentation for rails!

Like a number of rails enthusiats, I read Jeremy's blog post with a bit of trepidation. Mainly about the decline of rails blogging, it also points out the distinct lack of decent documentation for Rails... and firmly points a finger at all of us for not contributing to making it better. The worrying point is the conclusion that rails will decline in popularity due to a lack of available, comprehensive documentation.

Luckily, Rob Anderton has put my fears to rest. Rob's post has pointed me at the new Rails doc project. I've only just begun to investigate the site, but I can already tell it's far more useful than the standard Rails API I've been relying upon.

For one thing, it links the documentation to the version of rails you're using. This alone would have saved me a few hours of annoyance trying to get a method to accept parameters that are only available in a more current version than what's on our production server. :P

It also has a really neat keyword-search interface that prompts you with reasonable guesses, and a way for the community to add to the documentation (yay)!

All in all, it shows a lot of potential. So lets get cracking on building the doco up to a standard that developers in other languages have come to expect.

Sunday, 20 July 2008

One of Pradiptas children

Ok, so I put this into a comment on a previous post, but the phenomena has continued to spawn new children, so I thought I'd add to it.

Two days ago I opened my inbox to find it bursting with the results of an epic cascading thread. An overzealous recruiter had made a massive faux pas. He spammed 416 random rails-related folks with a Rails job offer... but put all their email addresses into the CC field.

The list ran the gamut of Rails, from those who'd barely touched it once, through the average rails developers (such as myself) right up to community luminaries such as DHH and Ryan Bates (of RailsCasts fame).

Reverse flash mob

After a few responses using reply-all, and the typical "don't use reply-all" (using reply-all), suddenly the mailing list became self-aware. Despite being accidentally gathered together in one place through an epic mishap, we recognised that we had a common interest... and a captive audience.

Most members of the group took advantage of this to chat about upcoming Rails conventions and what work we all did, and where.

Within two hours a google group (pradiptas-rolodex) was formed. Swiftly following were:

Sadly the wiki page keeps being deleted...

Anyway, it's been fun.

Highlights from the original thread

The post that began it all

I have a couple of Ruby on Rails position, wanted to know if you are
interested?


Max Archie
Technical Recruiter
Prodigus Source

The post that unified us as a community.

> PLEASE TAKE ME OFF THIS CRAZY EMAILING LIST!!!

Please KEEP ME ON THIS CRAZY LIST!!!

You all seem like pretty interesting folk, and perhaps serendipitously
we've all been added to the same ruby on rails marketing schloc. Even
more interesting to me, is that both my friend Harper and I are on it,
so hell they guy at least has good taste.

So in the spirit of fun email lists

Hi, I'm Anders Conbere, I Hack, run a co-working space in Seattle, and
love me some erlang and python.

What do you guys think of starting a google group I was thinking

"Igotfuckedinbanglalore"

~ Anders

P.S. I've thoughtfully removed our spammer from future communication

The best repsonse to the ad

Hi Max,

I am a recruiter who has an opening for a top-tier recruiter such as
yourself.  I need someone who can unwittingly set off the fury of *at least*
400 people, while ignoring all basic email etiquette.  Would you be
interested?   If not, do you know anyone else who is currently looking for
such an opportunity?

Sincerely,
Thanks for the mile long email thread out of freakin nowhere

Best representation of pradipta's mistake in code

PS:  Sorry for crashing anyone's iphone, but I couldn't resist replying.

module Recruiter
   class Email
      def initialize(site = "http://workingwithrails.com/")
          @recipients = Recruiter::EmailHarvester.harvest(site)
      end

      #
      # TODO: Really should change this to do a BCC instead of a TO
      # so 416 people don't get spammed and start an internet phenomenon of
      # replying to this message until the entire internet crashes.
      #
     def send_email(to_recipients, message_body =
File.read("really_bad_generic_email_body.txt") )
        @message = EmailMessage.new
        @message.to = to_recipients
        @message.body = message_body
        Mailer.send(@message)
     end

   end
end

Friday, 18 July 2008

Death by powerpoint

Check out this amazing powerpoint presentation* on making amazing powerpoint presentations!

* no, you don't often get to say that.

Thursday, 17 July 2008

Experienced Rails developer available - Sydney

Hi all, My contract is finally coming to a conclusion, so I find myself looking around for the next gig. I have a bit over three years full-time commercial experience with Rails, which I've been using commercially since just before it went 1.0. I'm looking for a contract in the Sydney area.

If you're interested, you can email me at home (24hr turnaround).

Tuesday, 15 July 2008

Little boxes, made of errors...

The standard styling for error boxes (at least with v1.2.6) looks ugly as per below.

Screenshot - old FieldWithErrors

The red doesn't go all the way around the actual box for form fields. Yuk! This is a result of the field in question going into a span that is styled - not the actual form field. Stylesheet code as below:

.fieldWithErrors {
  border: 2px solid red;
  background-color: red;
  padding: 2px;
  display: inline;
}

To make it look neat and pretty (as in the below image), use the following code instead:

.fieldWithErrors input, .fieldWithErrors textarea, .fieldWithErrors select  {
  border: 2px solid red;
  display: inline;
  padding: 2px;
}
Screenshot - new FieldWithErrors

Monday, 14 July 2008

Your customers *do* know what they want

I had a conversation with an old school friend last night; and we had a brief discussion about entrepreneurship as it relates to IT. He said he wanted to get out of the daily grind, and thought it'd be great to come up with a product and sell it. He was a bit leery, still, not sure about whether he could really "make it".

Having read a number of Paul Graham's fantastic articles I thought the idea was great, but I reminded him of how important it was to make something that your customers actually want. At which point he stated: "I speak to my customers all the time and in my experience they don't know what they want"

At the time I didn't answer. I could see he had a point, but it wasn't the one I was trying to make, and the conversation moved on. But now I think about it I think that he was simply wrong.

What question?

When you ask a user what content they want on their site or ask about styling or layout, you'll frequently bump up against a puzzled look and a lot of hemming and hawing as they try to figure it out. They don't have an answer ready for that and they probably didn't think about it before you asked. Often they have no idea what they want.

Many is the project where the requirements change like a fickle breeze as the scope creeps inexorably onward. It's obvious your customers don't know what they want. Right?

or do they? is it possible you're asking them the wrong questions?

What language?

Since I moved into freelancing, I've been studying up a whole lot more on business. One thing I was surprised to learn is that my customers are actually pretty savvy. They know their business, and they know exactly why they came to you to build their website.

Your customer knows they want a website that promotes their brand to get more customers, or to cross-fertilise their customer-base with their other product-lines, or to act as a tool that is a more efficient/less-expensive way of selling their products.

What they want has to do with their *business* goals, not with the specifics of the technology that will be used to achieve that goal. They know exactly what the goal is and whether they have reached it - and it has nothing to do with font-sizes or table layouts, or whether the "order" button should be on the product list-page or on the product's details-page.

If you don't know what your customer wants - it's most likely because you haven't asked them the right questions. But it's not enough to change what you're asking - you also have to change how you're asking it.

Your customer lives in a different world to you. You live in IT-world, but they live in the world of their business. Your customer uses a language that is different to IT language. They speak in the language of business - with market-penetration, conversion-rates and revenue. They don't operate in the realm of splash-pages, JavaScript or three-column-layouts. Business has its own language, its own way of organising and expressing ideas, and its own way of determining whether or not a set of actions will acheive a required result.

More than your job's worth

If you don't speak the language of business, then you'd better learn quick! These are your customers, and you already know they can't speak IT, and don't have the time or patience to learn. So it's up to you to do the translation service.

After all, that's your job. It's why you get paid the "big bucks" right?

It's your job to learn how to ask questions in such a way that your customer understands what you're asking and why. To translate their requirements into IT-jargon, and your questions back into business-speak. To express their choices in such a way that it gives them enough information to actually make the decision that you are asking of them.

Do they want flash or XHTML? does it matter to them if you use table or CSS-layouts?

No - what they want to know is "will this process of online ordering appeal more to my customers than the other, or will it be too difficult to learn?", "will my customers be able to find my products on this website, or will they get confused and I lose a customer?", "how can I use this software to give me better tracking of purchasing so I can promote or cross-promote appropriately?".

Tell them that if you place the "order now" button on the product-listing page, it could speed up the ordering process, at the expense of creating visual clutter. Explain that new users might get confused by this, but that it could aid frequent users to get through the process quicker. It then becomes a business decision: ie "are we aiming this site at new customers, or our returning/frequent customers?" - which is a much more meaningful basis for their decision-making.

So...

Translate your requirement questions into their language. Explain how their choice will impact them and their business goals. Explain the benefits in a way that they will understand. Make sure what you ask them leads back to their goals - not yours, and you'll find that they actually have a very good grasp of what they want.

Thursday, 10 July 2008

MySQL hates Timezones

We've started using the TzTime plugin which leverages the Tzinfo gem. It's pretty good. I've adjusted it slightly for my own preferences, but it works pretty well out of the box.

Unfortunately, we ran into some odd issues. Rails suddenly started to find records that it shouldn't. Eg if we were looking for all Event objects that occurred between two given dates, (and expecting to find none) suddenly we were finding one. Which played havoc with our tests :P

The occurred_at date on the Event object was clearly outside of the date-range we were looking at. We even tested to make sure the timezone wasn't stuffing it around and moving it into the date-window.

Eventually we discovered that the to_s(:db) command was printing the date out in a weird format using a T: '2008-07-10T00:50:23+00:00'. This is quite different from the standard MySQL format, which would have it as just: '2008-07-10 00:50:23'

MySQL needs to be set up properly to accept the "T-format" for DateTime objects and ours hasn't been. However, in our case it's much easier to simply extend Rails than to go through the required infrastructure politics to get global database settings changed (which will affect all the other projects in the business).

So I just updated the way that the to_s(:db) format works to always use the 'Y-M-D H:M:S' format as below.

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(:db => '%Y-%m-%d %H:%M:%S')
ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.merge!(:db => '%Y-%m-%d %H:%M:%S')

This seems to work just fine. It means that we'll need to update this if we ever change away from MySQL - but MySQL seems to be hard-coded into this organisation and seems pretty unlikely.

Note: when I first put this code into config/environment.rb it worked fine until I ran the tests - at which point I immediately got an error: uninitialized constant ActiveSupport. This went away the moment I moved the code below the Rails.initializer code (ie underneath the line that says: Include your application configuration below).

Monday, 7 July 2008

Not dead...

I'm not dead - just been In Thailand. I'll post again when I get up to speed.

Tuesday, 17 June 2008

assert email sent_to recipients

We have notifications sent out to users when their orders are finished processing. Recently we added "user groups" and so orders (which used to belong to a single user) now belong to groups of users. Which brought about the possibilty that a user could be removed from the group (and therefore could no longer see the order).

It's possible that this could happen between the time they submit their order and the time that the order has finished processing. We determined that, if the user no longer belongs to the group, they shouldn't get the notification - it should go to the next person in the group.

So our original notification code had something along the lines of:

  # Prepares an email that will notify the user that his calculation is complete
  def report_complete_notification(report)
    setup_common_values(report.user)
    @subject    += 'Your report is complete'
    body[:report] = report
  end

  # Details common to all emails from us
  def setup_common_values(user)
    recipients  user.email
    from        DEFAULT_FROM_EMAIL
    subject     "Our Funky Service - "
    sent_on     Time.now.utc
    body        :user => user
  end

So first we needed to update the notifier to call an appropriate 'recipient' if needed:

    setup_common_values(report.recipient)

Using the following recipient code in the model object:

  def recipient
    # always belongs to the original owner if no group is specified.
    return self.user unless self.group
    # use same recipient if creator still belongs to the request's group
    return self.user if self.group.users.include?(self.user)
    # otherwise grab the first person out of the group
    return self.group.users.first unless self.group.users.blank?
    # no users left - who do we notify?
    raise "HELP! The report: #{self.id} belongs to group: #{self.group.id}, but the group has no users so I don't know who should receive my notifications."
  end

As you can see, we aren't yet sure where to escalate notificaton if the group is completely empty. We have several options, but we're still waiting on the customer to get back to us and tell us what they want. The most likely option is to send an alternative notification to our admins.

How to test?

So now we need to test whether the appropriate person gets the notification. To begin with, testing whether an email went to a recipient, or set of them, isn't a pretty assertion, so I've abstracted it out into its own assertion (below). This also allows me to extend it if we need to check CC: and BCC: lists later. For now we don't use them, so "to" is good enough.

  # assert that the given recipients appeared in the recipient list of
  # the given email. If "want_match" is false - assert that *none* of the
  # given recipients were in the recipient list
  def assert_recipients(eml, recipients, want_match = true)
    recipients = [recips] unless recipients.respond_to?('[]') # arrayify if single item
    recipients.each do |recip|
      match = eml.to.any? {|r| r =~ /#{recip}/}
      match = !match unless want_match
      msg = want_match ? "email should have been sent to recipient #{recip}, but instead went to: #{eml.to}" : "email should not have been sent to recipient #{recip}"
      assert match, msg 
    end
  end
  # convenience function for asserting an email had recipients
  def assert_sent_to(eml, recips)
    assert_recipients eml, recips
  end
  # convenience function for asserting an email did not have recipients
  def assert_not_sent_to(eml, recips)
    assert_recipients eml, recips, false
  end

Now we can do various tests such as:

  # note - assume mailer.deliveries is instantiated as @emails in setup.
  def test_should_send_completion_notification
    order = orders(:completed)
    
    UserNotifier.deliver_completion_notification(order)
   
    assert_equal 1, @emails.size, "should have received one email to confirm completion"
    assert_sent_to @emails[0], order.user.email
  end
  def test_should_send_completion_notification_to_new_recipient_if_user_moved
    order = orders(:completed_for_moved_user)
    # sanity check
    assert_not_equal order.user, order.recipient, "user has moved group, but they are still showing up as the recipient"
    
    UserNotifier.deliver_completion_notification(order)
   
    assert_equal 1, @emails.size, "should have received one email to confirm completion"
    assert_sent_to @emails[0], order.recipient.email
    assert_not_sent_to @emails[0], order.user.email
  end

Monday, 9 June 2008

Rails Gotchas: Missing host error in unit tests

I finally got around to adding nice "and if you'd like to see your newly-created report, go to the URL: BLAH" links into my notification emails... a tiny nice-to-have that wasn't high on the priority list.

Suddenly my unit tests started spewing errors of the form:
ActionView::TemplateError: Missing host to link to! Please provide :host parameter or set default_url_options[:host]

The doco on default_url_options says you can add the "host" param on ActionController::Base, but no amount of monkey-patching in config/environment.rb seemed to do the trick.

Googling found a link talking about re-writing your URLs and it mentioned the problem is mainly seen in unit testing, and gave a couple of lines for a fix (as below)... but didn't say where to put them.

  include ActionController::UrlWriter
  default_url_options[:host] = 'localhost'

Experimentation showed that if you put them in your Notifer model (eg user_notifier.rb) they work just fine. Otherwise it just doesn't get into ActionMailer. :P