Showing posts with label article. Show all posts
Showing posts with label article. Show all posts

Tuesday, 24 August 2010

The two faces of Optimism

I've often been accused of being a pessimist.

I'm actually not. Far from it, I truly am an optimist. The problem seems to be that often people use the word "optimist" to mean something quite different to my (accepted) understanding of the word; and so, not agreeing with their worldview, they label me as a pessimist.

I think the real issue stems from divergent ways of viewing the term.

There are two definitions in common use.

A world of abundance

First is the one that I subscribe to. Optimism, for me, is to see a world of abundance, rather than a sea of troubles.

I believe that there is a solution to every problem (even if the solution is to throw out the old game and find new rules) and that anyone has the ability to get through the problems and find their dreams. Nobody is a lost cause and there is plenty of abundance to go around.

For me, that is optimism.

*You're* not an optimist

The other view I generally see in the pejorative sense. I can best explain by example:

You don't agree that my <* patently-impossible or highly-improbable *> plan will succeed, That's because you're just not an optimist!

I guess I'm just not a yes-man

Let me be clear: It's not "optimism" to be in deep denial about the efficacy of your plan, and I am not a "pessimist" just because I have looked at the numbers and found that they don't add up quite as well as what you hope them to be. I simply don't agree with you.

I am not a pessimist because I believe that *this* solution of yours won't work. I am an optimist because I believe that there *is* a solution, and that we will find it if we keep trying.

Optimism and employee relations

This issue often crops up at its most pathological in employee relations. In my experience, this is because there's a certain breed of manager that simply doesn't want to hear anything negative - even if it's the facts surrounding a potential solution. If solutions to a problem are proposed, then the most optimistic *sounding* solution will often win, regardless of actual likelihood of being the correct fit. Any nay-sayer to the offered solution will simply be dismissed as a curmudgeon, and thereafter will often be viewed as a "troublemaker".

As human folly goes, if the plan then fails, the nay-sayer is hardly likely to receive any recognition for having pointed out the potential... in fact, if the subject is ever brought up again, greater dislike is sure to follow - nobody likes being told "I told you so", especially on the heels of defeat.

More often than not, however, the initial solution will not fail miserably, but will simply provide sub-optimal results. The originator of the idea will be convinced that it was the best that could possibly have been achieved and that you were clearly a pessimist to think that it could fail...

Trust me... there is no point to trying to persuade them that your idea could have been better.

So, what to do?

The only way out of this dilemma is to make sure that instead of simply taking pot-shots at a bad idea, always provide an alternative solution that you can show is more likely to work. In this way you can show that you are optimistic about solving the problem, just not about the particular solution on the table. It's even better if you can reuse aspects of the original solution in your own - to show that the first proposal wasn't all bad, just needs re-alignment... but this isn't always possible.

It's certainly not a guaranteed solution. The first idea may well not be removed from the state of play - and may still be valued seeming more optimistic (ie providing a promise of a better end-solution... even if not more likely to succeed). This is even more likely to happen if the solution has been championed by a manager in an environment where Highest Paid Opinion wins reigns supreme). However, offering a solution will show that you are presenting a positive outlook, and you're less likely to be tarred as strongly with the pessimist/troublemaker brush. You show that at least you are trying to be helpful.

Of course... in some environments it still makes little difference, and even a dissenting opinion will brand you.

In this case, my suggestion is to be an optimist and trust that the world will get along in the end, and that perhaps there might be bigger, better places for you... if you choose to go looking for them. :)

Notes

Pessimist:

In reaction to them, I'll often tell them that I am a "realist" - which unfortunately makes them just snort and underscore the "pessimist" label in their heads. After all, I've found that a lot of optimists don't know the difference. Of course, mainly I'm just playing with their heads. :)

yes-man:

Sorry, "yes-woman" just sounds stupid, so I won't use it regardless of the truth of my actual sex; And I will bite anybody that suggests that I use "yes-person" :P

Tuesday, 29 June 2010

Software: no compensation necessary

I've just been reading "The World Is Flat" by Thomas L Friedman. It's an interesting book with lots good to say[1] about how the world is rapidly flattening - ie everywhere is becoming reachable from anywhere.

Along the way (and only incidentally to the main point) the author brings up the concept of Open Source Software.

He discusses the concept in a fairly positive light. He seems quite fascinated by the phenomenon - though doesn't seem to really get why people would do it. Though he makes a valiant effort to see things from the POV of the average OSS contributor, it's still fairly clear that he thinks it's a bit of a quaint, idealistic example of altruism (recognising the "pay it forward" aspect), rather than an effective and useful business technique. This is exemplified by the fact that he interviews a Microsoft Manager to ask whether or not it is sustainable as a business concept. Not bothering with an Open Source advocate, just an employee of one of the biggest proprietary systems out there.

One point stands out from what this employee says about OSS. He states that money is the main motivator for building software. From this the author concludes that any "really good software" will more likely be built by big software houses (eg Microsoft) than by OSS initiatives... after all, they have a lot of money, that can buy a lot of development time. He therefore concludes that The big software houses will always build the best software - simply because they can sustainably pay the most developers.

It's an interesting theory, and the most logical for those who don't really understand the motivation of the average OSS contributor. The reasoning behind this conclusion is of itself quite interesting. They claim that software is developed because businesses need stuff to be done - and thus they pay developers to do it. Nothing wrong with that... but they seem to mistake the effect for the cause and assume that the motivator for software to be built - is the money.

This is not the case.

Software is susceptible to the old phrase of "Necessity is the mother of invention" - ie the cause of software to be built is the business's need for it. The money is a secondary affector - ie the need causes the business to find money to make it happen. The originating need will be present regardless of whether the business has money to pay for developers... after all, even if the business has no money, the need will still be there for the software.

If the business happens to be made up of developers, then they are quite capable of creating the software, regardless of payment... and that is how you get startups: built and run by developers who see the need but don't necessarily have money to pay other developers - they just need the software to get their own business running, and are ready and willing to work in their own spare time to make it happen (nothing wrong with Sweat Equity).

The Open Source world calls this motivation "scratching your own itch" and it is recognised as the primary motivator for Open Source contribution[2]. Even for the really big projects.

Linux, for example, was Linus Torvalds' project to build himself an OS that didn't suck. Since he released it, thousands of other individuals have contributed to it for the same reason. It didn't do what they needed, so rather than bitching about it, they put their money where their mouth was and wrote the fix themselves... contributing it back to the OSS community... for no compensation except the "pay it forward" principle.

Which brings me to the other book I've just finished reading: "Drive" by Daniel Pink. One of the principle findings of this book is that fiscal compensation actually gets in the way of intrinsic motivation[3]. You can incentivise a developer to build something, but then they dissociate their ownership of the end product to the person doing the incentivising... care factor drops radically and you end up with... well... Microsoft products. Corporate and cold, buggy and with no real way to guarantee that the problems will ever get fixed.

This difference in motivation easily accounts for the fact that MS products are made primarily for creating more money for Microsoft (for which purpose they perform admirably), but only a passing interest in actually being fit for use: ie only so much as they can get away with before too many people stop purchasing them.

Now, there's nothing inherently wrong with that. More power to them for staying alive so long, and rather profitably - by keeping so many people satisfied for so long with what they're prepared to shell out... but there's a reason that I haven't used Microsoft products for over a decade now...

By comparison, the OSS world is driven purely by user requirements. If software is not fit for purpose, it simply won't be used... but more importantly, because there's nobody paying for it - if it doesn't do what it's supposed to, it will never be born

Conversely, if it is built, you can guarantee that it meets the requirements of at least *one* real customer (unlike many products built purely with potential revenue as a motivator).

OSS lives and dies on it's own merits... no compensation necessary.

Notes:

[1] Though I'd take a lot of it with a pinch of salt as well. This guy is clearly of the "yay capitalism, it solves all the world's ills" school of thought... though is slightly tempered by realising that humans are human, and some of them actually need supporting by those who are better-off... it's an interesting act of comparison to read this side-by-side with "The Shock Doctrine" by Naomi Klein - just as equally biased in the opposite direction, but also an fascinating read. I think if I ever put these two books together on a shelf, they'd annihilate one another with a very loud bang.

[2] If you're interested in this topic, "The Cathedral & the Bazaar" by Eric S. Raymond is an excellent book that explains the difference between what can be built from the ground up by need-motivated developers rather than the top-down software built by big software houses spending money to pay for what they want.

[3] which can be paraphrased as that innate curiosity and enjoyment of a challenge that occurs when you take on a new puzzle

Tuesday, 18 May 2010

Truth vs Harmony

People live on a spectrum of preference. Some prefer to hear the hard truth, regardless of whether it will hurt their feelings. Some prefer a comforting lie, feeling that harmony is more important than being right.

Both views have their pros and cons, and different situations generally call for a different balance on this particular scale. Still, people tend to have a preference for one or the other.

People often prefer different levels in different circumstances eg when we're more emotional, we tend towards comforting lie, and people will generally react to that - keeping any harsh truths for when a person is more able to bear it. Lying to children is often rationalised in this way - falsely in my opinion.

People at opposite ends of the spectrum often simply can't understand what the other people want. What motivates them to prefer a way of dealing with truth so different from their own?

Harmony people can't understand why a person would tell the truth if it might be more hurtful. They'll say, with perplexity "what they don't know won't hurt them, but the truth will hurt their feelings, so why do it?"

Truth people feel like they have no control if people keep lying to them, they'll ask: "They deserve to know the truth, after all, how could I make a decent decision if I can't trust what people say to me?"

Harmony people believe it doesn't matter as long as everyone gets along. Life is about getting-along, after all. Often-times, telling the truth causes them to lose face, which doesn't help anybody. But are they just trying to avoid fixing a problem that's their responsibility to fix? Are they avoiding a truth within themselves because they're too lazy or ashamed to face it?

Truth people feel robbed of a valuable chance to learn something if not told the truth. They feel that they are doing a service to another person by giving them a chance to see themselves and learn something. But are they just nitpicking, poking at wounds or being too lazy to be tactful?

In a bizarre twist, many people will say they prefer truth, but are actually lying. It's more comforting to believe you are a hard-hitting, strong person - able to handle any truth... it's much harder to face up to the truth that you are in fact not able to face up to the truth.

Personally, I've always preferred the truth. Yes, it can be hurtful. Yes - its embarrassing and sometimes degrading to know the negative effects of you impact on the lives of others, and there are definitely times and places where it's better to bite your tongue; but unless pressed, I'll always opt for the truth.

Both giving and receiving.

This can be hard for harmony-people to understand. Just as it's hard to explain that introversion[1] is not actually unhealthy, just a different way of being.

I'll usually get a response along the lines of: "but life is about enjoying yourself, not being right". Saying that I cannot enjoy myself if I know there's something I'm doing wrong, or even "It's just the way I am" generally doesn't help much. Harmony-people still often shake their heads and seem to think there's something unhealthy about the whole thing.

I've had more success when I explained that for me, improving myself is something I like to do. I work hard at it, and try to strip myself of self-delusions (though I'm under no delusion that I've got rid of them all). I am not on the quest for hedonism, but the quest for self-improvement...

So how does that fit with truth? Well - if I do something wrong, and you tell me it's ok - just to keep the peace... you're depriving me of my right to learn from my mistakes. That makes me feel upset.

I can appreciate that others prefer the opposite. They are trying to enjoy life and get through with a minimum of fuss and anguish. Most people seem to feel that if there's something nasty, it's better to coat it in shiny mother-of-pearl so that it's not all sharp and spiky in your side.

I can't help feeling, though, that it'd be much easier in the long run if they just faced up to it and spat it out to begin with. Then whatever it is would be gone from their lives and they could move on...

It often seems to me that people make their lives much more difficult by not facing up to hard truths, but by trying to ignore them or put them off. To convince themselves of the comforting lie in a misguided effort to "make it all go away". We all recognise these in other people... "I'm just big-boned", "I'll get around to it someday", "maybe it'll get better"... but when it comes to ourselves it's much harder, because we realise it's *us* that has to do the work to get past whatever it is... instead of somebody else's time/effort (which is as easy to spend as other people's money).

I am not immune to this myself, but whenever I find it happening in myself I tend to make an effort to getting rid of whatever it is I'm deluding myself about.

Obviously both can be abused. Lying to somebody can lead them into trouble, and telling too harsh a truth can indeed be needlessly hurtful - especially if a person is vulnerable, and simply needs comfort. Telling the truth about a person in front of other people can degrade their worth in the eyes of the others (and themselves). This is a Bad Thing.

Still, covering for another person's bad behaviour can mean that a person is not stopped from doing something they shouldn't - or never realises that they're being inappropriate (no, they don't always know!).

There is a place for truth and a place for tact, and it's a hard line to walk between the two.

IMO, though, society currently seems to place a heavy burden on the "play nice" aspect, and not enough on "keeping each other accountable".

The other argument about this is the lash-out defense of "it's none of your business". Who am I to tell other people to shape up? Especially since I'm so flawed myself. is this a "log in thine own eyes" situation?

To a certain extent this is the case. I certainly shouldn't feel high and mighty simply because I see a speck in your eye... but neither should I let you needlessly destroy your vision if you aren't aware of the issue.

It's absolutely my business to tell somebody that they need to learn to pick up the phone if leave me waiting for hours in the middle of nowhere, after they said they were going to pick me up. That situation doesn't have much of a grey area - but there are a lot of areas where it's more cloudy.

It's none of my business that a person is deluding themselves that their latest fad diet will work just as badly as all the others... unless they happen to be a friend that I care about and don't want to see them being unhealthy and eventually unhappy when their diet fails to work... again.

Is it my business? Yes and no. Should I find a tactful way of telling the truth? sure thing... but it will be the truth. The comforting lie will actually by more hurtful in the long run, to her health, to her self-esteem, to her chances of success at the very goal she has chosen for herself. In this sense, the comforting lie seems to only be less-hurtful in the short-term.

I believe this is true of most comforting lies. They cover-up a hurt to make life easier in the short-term, but the long-term harm is far more pronounced. On balance, I believe that only the truth will set you free.

Notes

[1] Introversion in the Myers-Briggs sense, an introvert is a person who regroups and regains their strength when they are by themself, rather than an extrovert who does so when in company. It is not actually the same thing as "being withdrawn" or even "being socially backward" - though these can be a consequence of too little interaction with others.

[2] Lying to children I think children are just as capable of bearing truths as adults, so long as they aren't frightening. Things that may make an adult fearful need not be so to children. eg death is a fact of life, but it needn't be something feared - of course you don't explain the scary details to a child as that will make it fearful.

Thursday, 6 August 2009

HyRes : ActiveResource that actually works!

It's three months on from my original post whinging about the lack of Validations in Active Resource.

At that time I put my money where my mouth was and forked a copy of the HyperactiveResource plugin, which provided a very crude, basic improvement over vanilla Active Resource.

So What have I achieved?

I've actually done a lot since then. I have implemented a lot of the TODOs that I wrote down as essential for us to get a good, solid basis for a workable system (see the done list below).

I'd class this plugin as working and functionally useful. Right now it's almost as useful as Active Record was a few years back... which is a vast improvement on the basic Active Resource interface.

I think there's still a lot of room for improvement, but the basics are there, and it *feels* like Active Record now - which before it definitely did not. Before it felt like you could play around with it for toy systems, or implement minor bits of functionality - but now you can really implement a fully-functional system on top of it. I know this - because we have done[1].

What's missing?

HyRes still has some niggling issues, and some stuff that would make it a much smoother migration for Active Record. I'm working on these and there is a perpetually-renewed TODO list on the HyRes README file

The funkier aspects of Active Record

I haven't yet re-implemented proper AR reflections - so the associations need some work... you can't do: widget.wodgets.find(...) and you can't use named_scope - which means that many basic plugins (eg restful_authentication) won't work out-of-the-box. But there's enough functionality there that pulling together a system that is totally non-Active Record is feasible.

API assumptions

Currently HyRes makes some assumptions about your API - it assumes a RESTful interface as the optimal configuration. If your API does not match the assumptions, there are some workaround available, but it might not be as useful.

An example is the validates_uniqueness_of method. This currently assumes that your API takes an array of conditions on the query-string, and that this will filter your returned set of findable objects.

If your API doesn't do this... the method currently defaults to fetching *all* objects and filtering on the rails-end... which is likely to be extremely slow (and may lead to timeouts). But it's there in case it's necessary. You may have to simply re-write that method with your own API-requirements in mind. I welcome alternative solutions to the problem...

testing...

Probably one of the nastier downsides atm is that HyRes doesn't have its *own* set of tests... We currently test it through the full suite of tests on our own system. This is mainly due to the fact that it's really quite hard to set up tests that don't rely on an existing API that's up and running. In our system we use a combination of mocks in our functional tests, backed by a fully-functional, running mock API for our integration tests... this is near-impossible for the HyRes plugin itself... so I'm still thinking about how I'll test it independently.

So what *does* it do?

The current list of implemented features (along with how to set up your API to use them) is also available on the HyRes README file but a quick snapshot of the list as of today is below. Note - in some cases, I still find it hard to believe that Active Resource didn't already implement these...

Columns

Because Active Resource isn't backed by a db... you can't use the table columns to determine the known set of attributes of a resource. ARes currently works by accepting any incoming data as an attribute, and using MethodMissing as an accessor for any known attribtues. This is fine for situations where you don't know what attributes will be returned by the remote system.

The downside is that if no value is returned for an attribute and you try to access it... it throws a MethodMissing exception (uuugly!).

If, however, you know what attributes to expect (because it's an agreed API-interface ), it'd be nice to be able to tell Active Resource which attributes we expect - and have it return any missing attributes an nil (rather than explosion).

Thus was born the columns= method.

Currently this method also works a little like attr_accessible - any attributes listed like this will be passed through on create/update... but nothing else will. This allows you to set temporary accessors (eg password and password_confirmation without them being passed over the wire.

Validations

directly pulls them in from ActiveRecord::Validations. For Rails 3.0 - we can use the ActiveModel::Validations component.

Note: validates_uniqueness_of is still experimental - as it requires a network-hit to actually determine whether you have any existing objects with the given field-value. You can't rely on uniqueness-of as there is no way of locking the remote-system's db - and thus somebody could have added a resource with that value in the meantime.

Callbacks

There are now callback hooks for before_validate, before_save etc - all the standard Active Record callbacks we've come to know and love. You can use them for callback chaining a la Active Record (still experimental - may have missed some, but you can definitely use validate :my_validation_method)

Finders

conditional finders

eg Widget.find(:all, :conditions => {:colour => 'blue'}, :limit => 5). This functionality working depends on your API accepting the above filter fields and returning something sensible. There's a lot of doco on how to set up your API (if you have that luxury) in the HyRes README

Dynamic finders and instantiators

eg find_by_X, find_all_by_X, find_by_X, find_last_by_X These rely on your API accepting filter fields as they are really just a bit of a convenient alias for find(:conditions => X). Dynamic finders take any number of arguments using _and_: find_by_X_and_Y_and_Z

We also have dynamic instantiators: find_or_create_by_X OR find_and_instantiate_by_X - both of which also take any number of args, just like the dynamic finders.

Dynamic finders/instantiators also take ! eg: find_or_create_by_X! will throw a ResourceNotValid exception if create fails

Other ActiveRecord::Base-like functions

  • update_attribute / update_attributes - actually exist now!
  • save! / update_attributes! / update_attribute! / create! - actually exist, now that we can do validation. They raise HyperactiveResource::ResourceNotSaved on failure to save/validate
  • ModelName.count (still experimental) - with optional finder-args. This works by first trying a /count request on your model object - and if the route fails it just pulls out all the objects and returns the length (ie - a nasty - but functional fallback).
  • updated collection_path that allows suffix_options as well as prefix_options = allows us to generate Rails-style named routes for collection-actions
  • no explosion for find/delete_all/destroy_all when none are to be found. (Active Record just returns nil - so should we)
  • ActiveRecord-like attributes= (updates rather than replaces)
  • ActiveRecord-like load that doesn't dup attributes (stores direct reference)
  • reload that does a full clear/fetch to reload (also clears associations cache, now that we have one)

Associations

A lot of the basic associations were in HyRes before I arrived - I've polished a few up (eg adding the belongs_to/has_many functions) and bugfixed

  • Resources can be associated with records
  • Records can be associated with records
  • Awareness of associations between resources: belongs_to, has_many, has_one (note - no HABTM yet)
    • Patient.new.name returns nil instead of MethodMissing
    • Patient.new.races returns [] instead of MethodMissing
    • pat = Patient.new; pat.address_id = 1; pat.address # returns the address object
  • Can fetch associations even with a nested route by using the ":nested" option on the nested resource's class. This command automatically adds a prefix-path, and will pre-populate the parent's id when you do an association collection_fetch.
  • Supports saving resources that :include other resources via:
    • Nested resource saving (creating a patient will create their associated addresses)
    • Mapping associations ([:address].id will serialize as :address_id)

What's next?

Well, one big thing that's next is that I'm planning on starting to work this stuff back into Rails itself. The brilliant component-architecture and upcoming Active Model changes for Rails 3 are just made for this. What I've been doing on HyRes can feed into that process to make Active Resource what we've always wanted - a real replacement-candidate for Active Model. For this purpose I have a fork of rails - but work will progress slowly (as I'm working on it in my spare time).

Until then, I'll keep on upgrading HyRes as needed for our own corporate requirements. Below, I've listed some features that Active Record provides and that I'd *personally* like to see implemented next... but this is certainly not an exhaustive list. (and I'm very open to other ideas...)

One thing I'd love to see is other people using HyRes for their own projects. I'd love to hear feedback on how it works (or not) as that will feed the project and feed into the work that eventually goes back into Rails. Better-still, if you're willing to work on implementing some of the still-missing features... you would be most welcome. Feel free to check out the Hyperactiveresource project on github and have a go.

Still TODO

  1. Testing - inside the plugin! (currently we test via our actual web-app's set of exhaustive tests)
  2. MyModel.with_scope/named_scope/default_scope
  3. MyModel.find(:include => ...)
  4. attr_protected/attr_accessible
  5. MyModel.calculate/average/minimum/maximum etc
  6. Reflections. There should be no reason why we can't re-use ActiveRecord-style reflections for our associations. They are not SQL-specific. This will also allow a lot more code to automatically Just Work (eg an Active Record could use has_many :through a HyRes)
  7. has_and_belongs_to_many
  8. Split HyRes into Base and other grouped functions as per AR
  9. validates_associated
  10. write_attribute that actually hits the remote API ???
  11. a default format for when it doesn't understand how to deal with given mime-formats? One which will just pass back the raw data and let you play with it?
  12. cache the raw (un-decoded) data onto the object so we don't have to do a second fetch? Or at least allow a universal attribute to be set that turns on caching

Notes

[1] The subject of an upcoming-blogpost will be on the work we've done: "Rewriting monolithic legacy systems in Rails"

Wednesday, 29 July 2009

CSV views with FasterCSV and csv_builder

We were asked to add a "Download as CSV" link to the top of each of our reporting actions. We already had a "revenue_report" controller-action with it's own html view template... and just wanted to add something similar in CSV.

FasterCSV seemed to provide some great CSV-generation functionality... but it builds the csv and spits it out right there in the controller.

uuuuugly!

We shouldn't put view-generation code into the controllers - that splits up our nicely-decoupled MVC stack!

I've also seen some ideas about putting a to_csv method into an individual model. Now, if I'm leery about putting view-generation in the controller... I'm even less impressed by putting it into the model! But I can see the benefits of simplicity.

However - I'm still not sure it fits with our requirements. A @widget.to_csv call works just fine for the WidgetsController#show action... and probably even the WidgetsController#index action (where you can run @widgets.map &:to_csv or similar)... but most of our reports span multiple associations and the data is grouped, filtered, and summarised. Each of these sets needs headings explaining what it is and laying it out nicely in a way that makes sense to the user reading the CSV file. Putting that kind of visual-only functionality in your model is where it really starts to get ugly again. I am left with one big thought:

Why can't I just put my CSV-template into the views directory?

enter: csv_builder plugin

csv_builder comes with extremely minimal documentation... but it's not that hard to use. Here's a step-by-step with examples to follow:

  1. install fastercsv and csv_builder
  2. add a .csv option to the respond_to part of your action
  3. add your <actionname>.csv.csvbuilder template into your views directory
  4. Add a "Download as CSV" link for your user

Step 1: Install FasterCSV & csv_builder

sudo gem install fastercsv
./script/plugin install git://github.com/econsultancy/csv_builder.git

Don't forget to restart your server!

csv_builder has already set up a format-extension handler for csv so you don't need to add it to the mime-types yourself, just start using it in your actions.

Step 2: In your controller action:

csv_builder will work out of the box even if you just add the simplest possible respond_to statement:

  respond_to |format| do
    format.html # revenue_report.html.erb
    format.xml  { render :xml => @payments }
    format.csv  # revenue_report.csv.csvbuilder
  end

With nothing more than the above, csv_builder will grab out your template and render it in csv-format to the user.

You can tweak the FasterCSV options by setting some @-params in your action. The csv_builder gem README explains how to use those, but the most useful would be to set @filename to give your users a meaningful filename for their csv-file. You can also specify things like the file-encoding, but go look at the gem README to find out the details

Here's an example of a full controller action:

# GET /sites/1/revenue_report
# GET /sites/1/revenue_report.xml
# GET /sites/1/revenue_report.csv
def revenue_report
  @site = Site.find(params[:id])
  @payments = @site.revenue_report_payments
  respond_to do |format|
    format.html # revenue_report.html.erb
    format.xml  { render :xml => @payments }
    format.csv do
      # timestamping your files is a nice idea if the user runs this action more than once...
      timestamp = Time.now.strftime('%Y-%m-%d_%H:%M:%S')
      # passing a meaningful filename is a nice touch
      @filename = "revenue_report_for_site_#{@site.to_param}_#{timestamp}.csv"
    end # revenue_report.csv.csvbuilder
  end
end

Step 3: add a template into your views directory

The final step is to add a .csv.csvbuilder template to your views directory. This is just like adding an html.erb template for your action, except that the file will have the extension .csv.csvbuilder. AFAIK csv_builder can only support template that have the same name as your action, so in my example the template would be called revenue_report.csv.csvbuilder

Your view template is where you can stash away all the FasterCSV guff that will build your csv file. This works the same way as any other FasterCSV-generated content - but just make sure you generate it into a magic array that is named 'csv'. Here's an example:

  # header row
  csv << ["Revenue report for site: #{@site.name}"]
  csv << [] # gap between header and data for visibility

  # header-row for the data
  csv << ["Date", "", "Amt (ex tax)", "Amt (inc tax)"]

  row_data = [] # for scoping
  @payments.each do |payment|
    row_data = [payment.created_at.to_s(:short)]     
    row_data << "" # column gap for visibility
    # note you can use the usual view-helpers
    row_data << number_to_currency(payment.amount_ex_tax)
    row_data << number_to_currency(payment.amount_inc_tax)

    # and don't forget to add the row to the csv
    csv << row_data.dup # it breaks if you don't dup
  end # each date in date-range
  csv << [] # gap for visbility

  # and the totals-row at the very bottom
  totals_data = ["Totals", ""] # don't forget the column-gap
  totals_data << @payments.sum &:amount_ex_tax
  totals_data << @payments.sum &:amount_inc_tax
  csv << totals_data

Step 4: add a CSV link

Something like the above code will generate a nice plain csv file and spit it out at the user in the correct encoding... but now we need something to actually let the user know that they can do this.

This is pretty simple - it just requires adding the format to your usual path-links. eg:

<%= link_to 'Download Report as CSV', site_revenue_report_path(:id => @site.to_param, :format => :csv) -%>

Testing

Testing for non-html formats still seems pretty crude. You have to manually insert the "accepts" header in the request before firing off the actual request - then manually check the response content_type. It'd be nice if we could just add :format => :csv to the request... anyway, here's a sample shoulda-style test case:

  context "CSV revenue_report" do
    setup do
      @request.accept = 'text/csv'
      get :revenue_report, :id => @site.to_param
    end

    # variables common to all formats of revenue-report
    should_assign_to :site
    should_assign_to :payments

    should_respond_with :success
    should "respond with csv" do
      assert_equal 'text/csv', @response.content_type
    end
  end # context - CSV for revenue report

Gotchas:

Don't forget to add config.gem!"

I fond I needed config.gem 'fastercsv' and a require 'csv_builder' in my environment.rb. You may need a gem-listing for both (esp if you're using bundler)

Rails 3 compatitbility?

The original csv_builder plugin is not rails 3 compatible and is in fact no longer being supported. But it has officially changed hands, to the newer csv_builder gem. This is Rails-3 compatible and not backwards compatible - though he maintains a 2.3.X branch for people to submit bugs.

csvbuilder or csv_builder

While the plugin is named csv_builder, be careful to name your files with the extension: csv.csvbuilder or you'll spend hours pulling your hair out about a Missing template error while it fruitlessly searches for your csv.csvbuilder file!

All your report data must be generated in the controller!

This makes you more MVC-compliant anyway, but if you had any collections being fetched or formatted in the view... now's the time to move them into the controller action as your CSV-view will need them too.

duplicate inserted arrays

You'll notice that the example template has a set of data being added to the csv array... you'll also notice that each row is generated on the fly - then I save a *duplicated* version into the csv array. If you don't duplicate it... you may do funky things with overwriting the row each time. In my case I also needed to declare the temporary array outside the loop in order to preserve scope. YMMV - I'd appreciate any Ruby-guru answer as to why this doesn't work without that.

don't reset csv

Don't do this: csv = [] it breaks everything!

Rendering as html

I've noticed there's a pathalogical condition where everything is going well - and it's even getting o the template... but it's *still8 rendering html.

I eventually figured out that it's actually rendering the layout around the csv... and it occurs when you've explicitly stated a layout (eg layout 'admin') somewhere at the top of your controller. To fix it, you just need an empty render without layout false eg

  respond_to do |format|
    format.csv do
      timestamp = Time.now.strftime('%Y-%m-%d_%H:%M:%S')
      @filename = "revenue_report_for_site_#{@site.to_param}_#{timestamp}.csv"
      render :layout => false  # add this to override a specifically-stated layout
    end # revenue_report.csv.csvbuilder
  end

Tuesday, 9 June 2009

acts_as_good_style

Originally published in 2007, this article is updated on a semi-regular basis and republished when I add something new...

Rails Idiom

There are a number of small pieces of rails idiom that contribute to better rails code. I've been collecting them at work as they come to me (often by reading other people's code). I know that a lot of this can be gleaned simply by reading the "Agile rails" book... but apparently this would actually take effort, and thus doesn't occur as frequently as I'd like. A lot more is just basic good common sense... but we all know how common that is. In any case, I'll probably put this up on my website soon and keep adding to it.

Note - I don't always have a justification for these things... as with everything - it's up to you to pick out the bits that work for you.

Basic good sense

Clarify your code

This is something that I learned from Joel Spolsky from his article on making wrong code look wrong. He suggests that code smells should stand out a mile off. Doing so will make it much less likely that we'll miss one and leave a hole for a bug to form in.

He further suggests that we should change the way we code to systematically remove all habits (even benign ones) that muddy the waters. If we make it really obvious what is right and what is potentially wrong - then it'll be easier for us to discriminate between the right and wrong bits of the code. This follows from basic psychology. By analogy: it's easier to notice black when all you can see is black or white - but adding many shades of grey makes it harder to detect. If we make a habit of writing only white (or pale-grey)... it's harder for us to miss a black spot.

Joel goes on to discuss a technique for using Hungarian Notation (in the way it was orignally intended) to increase visbility of safe/unsafe strings - which is meant to be just an example of one technique. (So don't take it as the main essence of his message).

Many of the things I bring up in this Idiom-Guide are techniques that draw from this philosophy.

Don't comment-out dead code

Don't comment out code that's no longer needed - just delete it. It'll still be there in the SVN repository if we ever need it again.
Dead code just clutters up the project and slowly makes it less readable/maintainable. :(

Avoid Magic global variables where at all possible

The "don't use magic globals" argument has been around for a while, but here's the rails take on it.

The Rails equivalent of a magic global variable is the "@variable" (though I've also seen some nasty uses of the params hash). I know that instance variables are essential for passing data from the controller to the view. But they should not be relied upon for anything else! eg do not set them and assume they exist when calling methods on a model - or in a library.

From a given piece of code that uses said variable, you cannot tell where the variable originates. If it were simply passed as a parameter to the method, there would be an obvious line of command. Instead, you have to trawl through all the earlier functionality to figure it out or resort to a recursive grep on the rails directory.

This becomes a problem if you have to maintain the code. Given a magic global variable: Can I delete the value? Can I change it without affecting later code (that relies magically on its value)? If I run across a bug whereby it comes in with a value I didn't expect - how do I find which method set the value incorrectly?

On balance - it's much better to simply pass an explicit parameter through the usual method-call interface, so at least you have a chance of tracing the line of command in future.

Don't add ticket/bug comments into the code

I don't wanna have to wade through hideous piles of comments referencing some obscure (fixed) ticket/bug such as:

# DRN: ticket ARC256 -- remove currency symbol
# end ticket ARC256

It wastes time and eyeball-space. This stuff can conveniently go into the SVN log message - where it'll be picked up by the appropriate ticket in the "subversion comments" section. This way it'll only get read by people who actually care about the ticket/bug in question.

Alternatively - what we get is a buildup of comments referencing random tickets that are probably no longer relevant. Especially because the ticket numbers mean nothing when taken outside the context of our ticketing system.
Think about it this way: how long would we keep that comment in the code? when does it get removed? whose job is it to remove the ticket comment or do they all just build up until we have more ticket-comments than code? How unmaintainable!

Please just don't do it.


Ruby idiom

Don't use .nil? unless you need to

There are very few times when you are specifically checking for actual nil results. Mostly you are just checking to see if an object has been populated, in which case use: if foo don't bother with if !foo.nil?
This also brings me to:

Use unless where appropriate

Don't check if !foo instead use: unless foo

This is just the ruby-ish way to do things, and it reads better in many circumstances... given that code is meant to be read by humans, this is a Good Thing.

Do not use and/or - instead use &&/||

The words "and" and "or" mean something quite different to the symbols: &&/||
Unless you understand *exactly* what you are doing - you should use the latter (&& / ||) instead.

Any expression that contains and/or is almost certainly inelegent and should probably be rewritten.

do-end vs {} blocks

{} should be used for single-line blocks and do-end for multi-line blocks... this is a "taste" thing - but it seems to be fairly common practice now.

Use single-line conditionals where possible

if you're only doing one thing based on a conditional, use: foo_something if x
rather than:

if x
   foo_something
end

Obviously this doesn't work if you're doing more than one thing or when you have an else-clause, but it's useful for reducing space eg:
return render(:action => :edit) unless @successful
which also brings me to:

Nested function calls need parentheses

In a nested function call eg :
return render :action => :edit
Rails can make a good guess at whether :action => :edit is the second parameter of the "return" function, or the first parameter of the "render" function... but it's not wise to rely on this and it will generate warnings. It's much better to parenthesise parameters to any nested function eg:
return render(:action => :edit)

However:

Don't parenthesise where unnecessary

When parentheses are not required for the sake of clarity, leave them out.
do_something foo, :bar => :baz


Basic Rails security

h( x-site-scripting )

In your view templates, surround all user-input data with the "h" function. This html-encodes anything within it - which will prevent cross-site scripting attack.

Do this even if the source is trusted!

If you get into the habit of making it part of the <%=h opening, it will just become natural. An example:

  <p> Name: <%=h @object.name -%>
  <br />Description: <%=h @object.descr -%>
  <br /><%= image_tag image_url_for(h @object.image.file_name) -%> 
  <br /><%= link_to "Edit object", edit_object_path(@object) -%>
  </p>

:conditions => ['protect_from = ?', sql_injection]

Most SQL queries are built up by Rails automatically. But adding conditions to your finds or your paginators brings in a slight, but real possibility of introducing sql-injection attacks.
To combat this, always pass everything in as variables. eg:

:conditions => ['name like ? AND manager_id = ? OR is_admin = TRUE', 
@user.name, @user.id]

Generators, Plugins and Gems... oh my!

Generators

use scaffold_resource

When you create a new model, generate a scaffold_resource, even if you aren't going to use the view pages.
It generates wonderful tests as well as your model and migration for free.

It's worth it!

It's also worth it to go through the hassle of entering all of the attributes on the command-line. Again because it generates tests, fixtures, migrations, views etc for you.
For usage: script/generate scaffold_resource --help

-x and -c

If you pass the "-c" flag to a generator (script/generate whatever), it will automatically svn add the generated files. This isn't necessary, but it's a really useful thing so you don't have to hand-add every file individually.

There is a similar flag (-x) for when you install plugins. This will automatically add the plugin to the project via svn:externals.

Plugins

Don't use a plugin unless it does exactly what you want

It's so easy to write functionality in Rails that there's no point in using somebody else's plugin unless it does *exactly* what you want it to do.
Especially don't bother using a plugin if you have to hack it to pieces and pull out half the useful functionality. You are just creating a maintenance nightmare in your future.

That being said, if a plugin has half the functionality you need... it's a great place to learn the bits you need to use.


DB and migrations

Use migrations

But you weren't going to hand-hack the db anyway were you?
If you don't use migrations, then capistrano can't deploy the changes automatically. That only means more work for you - and means we can't auto roll-back if something horrible goes wrong.

Don't go back and edit old migrations!

I now have a policy that states "migrations only go forwards". This comes from several bad experiences where a developer (sometimes me) has gone back to edit an old migration and totally locked up the system for everybody else.

You may know that a deployment requires you to "migrate down then back up again". But sometimes that either gets forgotten, or is actually harder to do in practice than in theory eg if you're using capistrano to auto-deploy your site, it's harder to get it to do twisty migrations than just to "move forwards one".

Besides, which, it's a hassle for you to tell everyone on the dev team "before you do an svn up, migrate down to VERSION X, then svn up, then migrate back up again"... what if somebody doesn't want to lose their existing data? what if they're an svn-junkie and have already done an svn up before you get to them? what if somebody is not in today and you have to catch them before they come in tomorrow?

It's way too easy to get your bodgied-up back-hacked migrations into a wedged state where a new person cannot back up or go forward, and often cannot even start from scratch (eg from a fresh copy of a database). This becomes a serious problem for you if you ever have to change servers (or set up a new testing server or whatever) and have to run all the migrations from 0 to get the db ready.

Even if it all runs as you expect, it's three extra steps to remember than just "rake db:migrate", which raises the spectre of Human Error.

It's annoying to make it work, prone to error and bloody awful to fix when it breaks. All in all much easier just to write a new migration to change the existing stuff.

Extra migrations don't actually hurt you - they really don't take up that much more space in your SVN repository - they're worth having them around for the knowledge that anyone, anywhere will have a completely up-to-date system even if they start from complete scratch.

Add messages to data migrations

All of the basic ActiveRecord migrations tend to spit out their own status messages. Data migrations don't, and it's often useful to know what your migration is doing - especially if you've got a long data migration that would otherwise sit there in silence, looking like it's hung your machine.
So drop in a few status messages eg say "migrated #{orders.size} orders, moved widgets over to new wodget style"

Don't add excessive status messages to migrations

Obviously the above messaging can go too far. Don't write out a line for every piece of data you migrate or you'll get heaps of scrolling text rather than an idea of which migration we're up to - that useful data can get lost in the noise. Just a status message showing how much has been done or what stage we're up to in the migration will do.

Use "say" in migrations

Instead of using "p" use "say" - it gets formatted nicely.


MVC-specific idiom

If you are unsure about what MVC is all about - I suggest you do a bit of basic research on the subject. It's really important to keeping the code clean in a Rails site. Rails is completely founded onthe principle of keeping the MVC layers separate - if you begin tying them together, your run the risk of causing the code to become brittle - losing the flexibility that is the reason why you're using Rails in the first place.

Basic MVC principles

Keep view code out of the model

Your model should be independant of your views. A widget doesn't need to know whether it's being displayed as html, XML or via a screen-reader. It is an independantly-addressable object that contains "data and some knowledge of how to maniupulate that data".

Despite this I consistently see things like methods named "show_edit_link" which apparently tell the html whether this model's edit link should be visible on the html list page. That sort of logic should be in the view only. The model can have methods that, say, determine if they should be "editable" by the current user... and the code that determines if the edit link is displayed could check that method, but should remain inside the view.

If there is common code in your views - then it can be pulled into the helper file for that model. If it's used across multiple models, then it should go in application_helper.rb. It should never appear in the model.

The other common booboo I see are "display" functions that include html (eg "format field X to display nicely"). This is also very wrong! It means that, later, when you want to change html styling across the entire site you have to dig around inside every file in the site to change it... rather than simply changing the views - which is How It Should Be. From having to do just such a nightmareish site-upgrade, I can tell you that the fewer files you need to change - the easier your life will be in the long run.

All view code should remain in the views - with commonality pulled out into helpers.

Data manipulation code should go in the models - not views or controllers

Ok, so you have a report that you want to display that pulls data out of your widget... and your report view pulls it out bit by bit and does some calculations on it - then displays it.

So what happens when your client asks you to now do that report in PDF format... or CSV as well?

If all your data-manipulation methods are on the model itself, then you only need to pass the model object into the view and the view can then pull the data out in the same way for each view style . The alternative is to have a messy set of views/controllers where the same code is duplicated, violating the DRY principle and adding to your future maintenance headaches.

All data-manipulation code should remain on the model that owns the data.

Model idiom

Group relationship commands at top of model

One of the most important and useful things to know about a model is what it is related to - eg the "belongs_to"/"has_many" (etc) relationships.
If these calls get scattered throughout the model object, they will get lost amongst the noise and it will be hard to tell whather a given relationship has been declared...
Thus it's Good Practice to group them all together - and to put them right at the "top" of the class. eg:

class Widget < ActiveRecord::Base
  belongs_to :widget_class
  has_many :widget_parts
  has_one :pricing_plan

  #... rest of the model goes here
end

IMO this goes also for composition objects, "acts_as_<whatever>" declarations and auto_scope declarations - all of which declare the "shape" of the model and its relationships with other objects.

Group validations

The second-most important group of information about a model is what consists of valid data for the model. This will determine whether or not the data is accepted or rejected by ActiveRecord - so it's useful to know where to find it. So again, do not scatter it all about the model - put it near the top... preferably just below the "relationship" data. EG:

class Widget < ActiveRecord::Base
  belongs_to :widget_class
  has_many :widget_parts
  has_one :pricing_plan

  validates_presence_of :widget_class, :pricing_plan
  validates_numericality_of :catalogue_number, :integer_only => true
  def validate
     errors.add_to_base "should have at least one part" if widget_parts.blank?
  end

  #... rest of the model goes here
end

Validations should obey the DRY principle...

Please save my eyeballs from having to read the same thing over and over, and obey the DRY principle. You can write:

# please use:
validates_presence_of :field_one, :field_two, :field_three, :field_four
# instead of:
validates_presence_of :field_one
validates_presence_of :field_two
validates_presence_of :field_three
validates_presence_of :field_four

You can also use "validates_each" if you have several things that perform ostensibly the same validation... but it isn't an included one. eg:

# please use:
validates_each [:field_one, :field_two, 
                :field_three, :field_four]  do |model, attr, val|
  model.errors.add(attr, "should be a positive number") unless val.nil? || 0 <= val
end
# instead of:
def validate
  errors.add(:field_one, " should be a positive number")  unless field_one.nil? || 0 <= field_one
  errors.add(:field_two, " should be a positive number")  unless field_two? || 0 <= field_two
  errors.add(:field_three, " should be a positive number")  unless field_three? || 0 <= field_three
  errors.add(:field_four, " should be a positive number")  unless field_four? || 0 <= field_four
end

Read the section on validations in the book!!!

There are lots of really useful standard validations available in Rails... PLEASE check for whether one exists before writing your own hack into "def validate"... you might be surprised at how many are provided absolutely free of charge!
One of the most useful being "validates_each"... so you don't have to repeat the same thing over and over and over... remember that DRY principle?[1]

Another is "validates_format_of" which allows you to perform several validations at once - and give a consistant error message eg checking if a percentage falls within the range (0-100) with an optional percentage sign:[2]

  validates_inclusion_of [:my_percentage, :your_percentage], :in => 0..100, 
                       :message => "should be a percentage (0-100)"
  validates_format_of [:my_percentage, :your_percentage], :with => /^(\d{1,3})%?/, 
                       :message => "should be a percentage (0-100)"

If a validation doesn't refer to a single field... put it on base.

When it refers to more than one field, feel free to use

errors.add_to_base <whatever>
don't do something silly like: errors.add(nil, ...)
That being said - if you can squeeze it onto a field, please do - as it really helps the user to quickly find where the problem is in their form.

Give the poor users some useful feedback when validating

Don't tell them "Your weights should add up to 100"... tell them "your weights add up to 82%, your weights need to add up to 100% to be valid". It actually helps them see exactly what's missing.

When you update your validations, update your fixture data

This one has got me a couple of times. The problem is that fixtures are loaded into the database without going through validations - so your test data could be wrong - even when rake test passes flawlessly :(

Name your callbacks and declare them as handlers

If you have a number of callbacks (eg before_save/after_create hooks), it's better if you name them as a method, and just refer to them near the top of the model. eg:

# instead of:
class Widget < ActiveRecord::Base
  belongs_to :widget_class
  has_many :widget_parts
  has_one :pricing_plan

  validates_presence_of :widget_class, :pricing_plan
  def before_save
     # adds default pricing if none was specified
     pricing_plan ||= PricingPlan.find_default
  end
  #rest of model here
end
# try using:
class Widget < ActiveRecord::Base
  belongs_to :widget_class
  has_many :widget_parts
  has_one :pricing_plan

  validates_presence_of :widget_class, :pricing_plan
  before_save :add_default_pricing

  # handler to add default pricing if none specified
  def add_default_pricing
     pricing_plan ||= PricingPlan.find_default
  end
  #rest of model here
end

This example is too simple to easily show it, but there are a number of reasons why this is a good idea.

  • Firstly, you've named the method, which gives you an idea of what it's for = easier to maintain.
  • Secondly, if you need to add multiple of the same hook, you don't need to mash them all into the same callback method - which also aids readability.
  • Thirdly, by keeping each call as a single line, you can list them all near the top of the model class without taking up heaps of room (especially annoying in the case that the methods start getting really long). This helps you to quickly see what is and is not being done, without having to read through the details of the methods... or, even worse, seeing the first "after_create" hook and thinking it is the only one, when another is hidden later in the class.
  • Fourthly, if the method ever gets more complicated, or has to change, your method is encapsulated - the call to the method is independant of any other callback handlers of the same type.

Use Rails helpers for simple find calls

Rails comes with lots of helpful convenience functions. Read the book and learn about them! The find helpers are a good example. Something like the following is an excellent candidate.

  # instead of
  thing = Widget.find(:first, :conditions => "name='wodget'")
  # use the find_by_<whatever> method
  thing = Widget.find_by_name('wodget')

It shortens the code and makes it clear exactly what you're trying to find.

View idiom

Use -%>

Make sure you put a minus sign before closing your %>. This reduces whitespace in the resulting html - which makes it easier for us to read when we need to diagnose a display problem. eg use

<%= render :partial => :foo -%>
<% if some_condition -%>
   <div id="bar">
       <%= bar_fields -%>
  </div>
<% end -%>

h( x-site-scripting )

See this topic in the RailsSecurity section under XSiteScripting

Apply the DRY principle

Templates can have common code too - even across resource-boundaries.
Eg pagination links, or a common header/footer. Common view code should be factored out into a partial template and can be shared by a single resource-type, or across multiple resources.

Shared partials go in /app/views/shared

If a partial is to be used in a common layout, or across multiple resources (eg a common header for all resources reached after login), then the partial should be stored under the "/app/views/shared" directory.

Suppose you have a partial named "_common_footer.rhtml" in the shared directory. You can access it by referring to the partial thus:

<%= render :partial => 'shared/common_footer' -%>

Don't write a string inside a code snippet

This is silly: <%= " | " -%>
In a template, you can just type the pipe (or whatever) without the extra FrouFrou around it.

Controller idiom

Never, never, never use @params or @flash

These were deprecated some time ago. Instead use the similarly named methods: params or flash
Incidentally - you should be reading and taking heed of the warnings that the script/server is giving you about these things...

redirect when moving on

When you are moving on to another stage in the process, you should use redirect. eg you have successfully updated a page and are now moving to the index page.
When you are coming back to the same page, you can use render (eg the user entered an error and you are redisplaying the 'edit' page).

The reason we do this is that a redirect will change the URL in the address bar to the new page. If you just render a different page-action, the user will still have the URL of the previous page, even though they are now looking at a different page. This is Bad because if the previous page was, say, a form-POST... and the user clicks "refresh"... Bad Things will happen. Whereas if you redirect, then the user will have the URL of the new page and it's perfectly fine for them to hit refresh to get the page they're on.

Make private methods obvious!

Private controller methods aren't visible as actions. It can be a problem if somebody doesn't notice your little word "private" and starts to add methods below it without realising - and then wonders why Rails can't find their shiny new action.

So make sure that it's obvious and difficult to make the mistake.

Firstly, put private methods only at the bottom of the file!
Why bother going through spaghetti code with continual calls to private then public then private again... etc and having to figure out exactly what level of visibility we are right now? sort them and put the private ones all together at the bottom.

Secondly, use a line like this:

    private ###################################################################################

To make it painfully obvious to all who follow that below this line thar be dragons...


Testing idiom

All testing

Learn your basic assertions

Another "please read the book" request - there are lots of useful assertions already written for you. Using the appropriate ones makes your life easier as the error messages are more appropriate/meaningful for your test case. I especially find it irritating to see things like assert foo == bar when assert_equal not only has a better error message, but also means you're unlikely ever to accidentally use only one '=' - causing the test to silently pass when it should have failed.

Put the expected value first in assert_equals

The error message reads better that way.

Don't cram multiple tests into a single test case

Do not write generic tests with a dozen things to test all in a single test case.
Example shown below. Note - the real code example I took this from actually had another 6 tests crammed into it...

There are (at least) two reasons for not doing this:

Firstly - if the first test here breaks... it doesn't keep running the other tests. If it breaks, you can only tell that the first test isn't working... if you split them out into individual tests, rake will keep running the other tests - this is extremely helpful as you will then know whether all the validations are failing - or just that one.

Secondly - naming tests to mean what they're testing actually helps you find out what's going on when it breaks. A generic "test all sorts of stuff" name doesn't help you much, but a "test that widgets can't be created if the size is negative" does. You won't be thinking about this when you're writing that test case (and are fully immersed in the code you're testing), but six months later when something breaks your test... you will want to know what it's actually breaking and every bit of help (no matter how small) counts.

# don't use:
  def test_create_should_not_accept_invalid_parameters
    assert_no_difference Widget, :count do
        wid = create_widget(:vat_percentage => -1)
        assert wid.errors.on(:vat_percentage)
    end

    # invalid price range
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_min => -1)
        assert wid.errors.on(:price_range_min)
    end
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_max => -1)
        assert wid.errors.on(:price_range_max)
    end
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_max => 100, :price_range_min=>200)
        assert wid.errors.on(:price_range_max)
    end

    # invalid size range
    assert_no_difference Widget, :count do
        wid = create_widget(:size_min => -1)
        assert wid.errors.on(:size_min)
    end
    assert_no_difference Widget, :count do
        wid = create_widget(:size_max => -1)
        assert wid.errors.on(:size_max)
    end
    assert_no_difference Widget, :count do
        wid = create_widget(:size_max => 100, :size_min=>200)
        assert wid.errors.on(:size_max)
    end
  end
# instead try:
  def test_create_should_fail_on_negative_percentage
    assert_no_difference Widget, :count do
        wid = create_widget(:vat_percentage => -1)
        assert wid.errors.on(:vat_percentage)
    end
  end
  def test_create_should_fail_on_negative_min_price
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_min => -1)
        assert wid.errors.on(:price_range_min)
    end
  end
  def test_create_should_fail_on_negative_max_price
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_max => -1)
        assert wid.errors.on(:price_range_max)
    end
  end
  def test_create_should_fail_on_backwards_price_range
    assert_no_difference Widget, :count do
        wid = create_widget(:price_range_max => 100, :price_range_min=>200)
        assert wid.errors.on(:price_range_max)
    end
  end
  def test_create_should_fail_on_negative_min_size
    assert_no_difference Widget, :count do
        wid = create_widget(:size_min => -1)
        assert wid.errors.on(:size_min)
    end
  end
  def test_create_should_fail_on_negative_max_size
    assert_no_difference Widget, :count do
        wid = create_widget(:size_max => -1)
        assert wid.errors.on(:size_max)
    end
  end
  def test_create_should_fail_on_backwards_size_range
    assert_no_difference Widget, :count do
        wid = create_widget(:size_max => 100, :size_min=>200)
        assert wid.errors.on(:size_max)
    end
  end

Group tests in a meaningful way

Test files start out with only a handful of tests... but this rapidly changes. Very soon we get vast numbers of tests covering all sorts of actions.
To reduce the potential maintenance nightmare, group related tests together and put in a comment to indicate why they are grouped together. Any new test should be either put with related groupings or a new grouping created for it.

As an example: the most common set of tests revolves around resource CRUD - in which case it makes sense to group tests with related CRUD actions (eg "edit" and "update") near one another.
Another example is for the "user" object. users go through stages (process of creating a new user, the activation process, CRUD for an ordinary/active user, then destroying/reviving a user), so tests in these stages are grouped together.

If you're using rspec or shoulda, you are provided with a natural grouping mechanism (describe and context respectively). Use that to your advantage and make your future life easier.

Use named fixture records

Don't use bare fixture ids in your tests (eg get :edit :id => 1), instead use named references (eg @quentin.id).

This is a bit of a gotcha as the automatically-generated tests often use bare ids. However, these should be avoided where at all possible.

There are three main benefits:
Firstly, it makes the code more readable, with fewer magic numbers and more meaningful variable names.
Secondly, if we must reorder the fixture records (and thus change the ids) - at least we know which record we really wanted and we don't need to scour the tests looking for the ids to change.
Thirdly - if you're using auto-generated ids (rather than specifically passing them in) you won't know the id - and assuming that it's 1 sets you up to fail.

Similarly, don't use magic strings - especially for things like testing login/authentication. This will eventually bite you. It's much better to specify @quentin.login, @quentin.email (or whatever) so you don't have to go through your tests updating at a later time.

Don't forget to reload

The number of times I've been caught out by this... when you change something and are testing that it got saved into the database correctly, make sure you use @foo.reload. It's especially important if you're checking that you get the right validation errors on your ActiveRecord... you then want to test that it *didn't* change in the database too.

Common functions can be made into methods

If you find yourself performing the same action over and over, don't forget that you can DRY it up by putting it into a common function at the bottom of the test file.

  def setup
    @default_widget_hash = {:name => 'My widget', :size => 42, :wodget => wodgets(:my_wodget)}
  end
  def test_should_create_widget
    assert_difference Widget, :count do
      w = create_widget
      assert !w.new_record?, "should not have errors: #{w.errors.full_messages.to_sentence}"
    end
  end
  def test_widget_should_require_name_on_create
    assert_no_difference Widget, :count do
      w = create_widget(:name => nil)
      assert w.errors.on(:name), "Should have required name"
    end
  end
  def create_widget(options = {})
    Widget.create(@default_widget_hash.merge(options))
  end

Really common tests can be made into assertions

If you find yourself testing the same thing in multiple unit/functional tests, you can DRY that up too it up by putting it into the test helper as a new assertion. Don't forget to name it "assert_X" (where X is descriptive).

Make sure you name it starting with "assert_". This convention allows you to see which methods will have assertions in them (and thus possibly cause your test to fail), as opposed to general methods that simply set-up/tear-down information.

  # Assert that the handed object (thing) has no errors on it.
  def assert_no_errors(thing, msg = "")
    fail("should have passed an object") unless thing
    e = thing.errors.full_messages
    assert e.empty?, "Should not have error: #{e.to_sentence} #{msg}"
  end

Unit tests

Test your CRUD actions

It may seem like overkill to test CRUD here *and* in functional tests, but this is the place to test validations and any things that will affect basic CRUD-actions (eg if you have a method that stops your Widget from being deleted after it's been submitted for approval... then you need to try deleting a submitted one and make sure it fails!

If you only leave it up to your controller tests to check this stuff, then you are getting your data back filtered through one more layer of abstraction and will have to sort out whether it's a model-bug or a controller-bug. It's better to be sure of your model before adding an extra layer on top. It really makes your life easier in the long run!

Sadly scaffold_resource no longer writes these tests for you. :P

Test your validations

Each validation should be tested by trying to create your model object with that one set to nil. Trust me, it's worth it in the long run!

Functional tests

Test your validations

Create a functional test for each validation - again, try to create an object, nulling out the attribute-under-test. I know this seems to double-up on the unit tests - but what you're testing here is that the model is correctly getting back an error on the required attribute. This will be what is displayed on the user's screen - so they can fix the error before moving on. You also need to test that the template is still on the "edit" page. If it's not, then all the ActiveRecord Errors will be in vain as your user will never get to see them.

Test authentication-levels

If you have different levels of authentication (eg administrators get to see everyone's stuff, but you can only see your own stuff) then you must test both ways... test that you can see what you should see, but also test that you can't see stuff that you shouldn't see! You don't want your users peeking into your admin accounts just because you missed the "unhappy path" tests.
...and don't forget to test what happens when you're not logged on too.

Tests need more than just "assert_response :success"

Unfortunately, "assert_response :success" only checks that *something* happened - not that it was the right something.

To be more precise, "assert_response :success" asserts that rails successfully returned an "HTTP 200" response (as opposed to a 404 or a redirect). Unfortunately it does not check that you received the response you were expecting... (though I wish it were that simple) - sadly you do have to actually check that it not only returned a response - but that it was the correct response with the correct variables etc.

In the case of a "get" request - you should check that we are rendering the correct template, and that the "assigns" variables have been correctly populated.
In the case of a request that does something - check all the variables afterwards to make sure they changed in the way that you expected (eg that the variables actually were updated).


Notes

[1] Why is it that the DRY principle is the one I seem to have to harp on the most? Repeated endlessly reaching a pinnacle of irony.

[2] check my earlier post on "validates_percentage_of" for this example in context.

Tuesday, 28 October 2008

Headhunters strike out again

A follow-on from my negative experience with headhunters. I've begun receiving exactly the sort of "job offers" that I was fearing. I've got offers (from all over the world, mind) asking me if I'd like to work as (for example) a junior developer on an embedded-kernel contract in Belgium... which'd be nice and useful if I were interested in that sort of thing instead of having only 6 months non-commercial experience (at uni) doing this stuff as a elective subject... or at least held a work-permit that allowed me to work in Belgium.

The end result makes it fairly obvious that their recruiters are just too damn lazy to do more than a quick search on the db... followed by spamming all the results. They've ripped out the db-friendly data (ie my naked skillset) and divorced it from the important stuff (ie what sort of job I'm actually interested in). Leaving me to be spammed by every idiot with a contract to push.

If I were a junior developer, desperate for any job I could get, I'd probably just have to suck it up... but I'm not - and they're alienating an in-demand developer with lots of experience. This doesn't make me feel like responding positively to any of their future offers... even if they do suit my requirements.

What gets me is that it'd be so easy to make this system work. Even if they had a few extra fields in their db for "what type of job is the candidate looking for", they could run these as a filter over their resultset and turn that spam into much-less-despised targetted advertising.

I'd even be interested (possibly even grateful) to receive such a service - instead of so annoyed that I've blogged about it twice.

Sunday, 28 September 2008

Headhunters and a dearth of trust

What is it with some headhunters?
I had a phonecall from a headhunter the other day that really rubbed my fur up the wrong way.

First, they wanted my CV in Word format. I hate that. Quite apart from having to explain that I use Linux - which frequently leads in to the "what is an OS" discussion that I find so annoying coming from an IT recruiter. Yes I can use OpenOffice to create a Word doc, but it's a preference thing. I prefer to write my CV in PDF so I have more control over the formatting - because I've found that .doc format "helps" too much, and suffers from bit-rot.

But it's not like they can't read/print-out a copy of my CV if it's in PDF. What really gets me is that there are only two reasons for wanting my CV in Word. First is so they can copy/paste my skills/experience into their mammoth database - reducing me to a set of numbers... which I really don't like.

I recognise that might be a knock to their usual system, but I really don't like being just a number in an uncharted sea of applicants. It doesn't work for me to be recruited by a company that would view me as only so important as the number (and not the quality) of the years of experience I have.

Second is that they want to remove my contact information so that they are the only point of contact between me and the employer. This view is especially reinforced by headhunters that refuse to give you the prospective employer's name, industry, size or any other potentially-identifying details (or politely change subject when I ask, as this one did). How the hell am I supposed to decide if I'm interested in working for a company I know nothing about?

I should point out - I get a phonecall a week from various recruiters. I don't have time to go to an interview a week - so I have to decide, based on what you tell me, if you are worth a morning of my time.

Word to the wise: recruiters perform a valuable service - both to employers and employees. If you are good at what you do, then nobody will be unhappy about paying you for your services! The only recruiters that need to resort to cheap tactics like hiding names are the ones that cannot build a reputation for excellence on their own... thus if you flat-out refuse, then it instantly gets my suspicions up.

Yes, I'm sure you will get ocasionally burnt by employees and employers that are unscrupulous and try to screw you out of your fee. But are those the kind of clients you want to keep anyway?

Recruiting is a people business - it is outsourced HR and builds upon long-term relationships. Trust is important, and screwing around like this ruffles that layer of trust and just feels tacky.

Please don't do it.

Saturday, 26 July 2008

Best vs Good enough

I can't remember which I've heard - I think it was "good enough is the enemy of best", but it could as easily be the other way around. It's a bit confusing as I've definitely heard conflicting stories.

We are the best!

From one perspective, you want your new startup to be the best - because, in a lot of ways, that is what will distinguish you from your competitors. The difference in market-share between Google and Yahoo! is an obvious example. So is the difference between Amazon and... whoever comes second after Amazon.

Clearly it pays big to be the best in your niche. You need to be the best. "Good enough" not only won't cut it, but will relegate you as indistinguishable from the swamp of mediocrity in which all the other companies float.

In this sense, "good enough" is the enemy of "best".

Good enough. Ship it!

And yet, you need to *ship*, because if you don't have anything to ship, then the customers will never hear of you. This follows from the principle of "ship early, ship often". It's all too easy to get caught in analysis paralysis, endlessly perfecting what you've got without ever putting it "out there" for real users to get their hands on.

In that respect, "best" is the enemy of "good enough".

So which is it?

As with everything, you need a strategic balance.

You don't need to be best at *everything*. You'll wear yourself out trying. You just need to be best at what you've chosen for your market differentiator. Perhaps you're aiming to provide the best features for power users, or most comprehensible features for beginners, or even the nicest customer service. Whatever you decide on, use that focus to drive your quest for perfection. You can push for the best in this one area and let other areas be "good enough".

If you try to swallow the elephant whole, you'll never get it down. Instead, keep focussed on your core competencies, and don't worry *too* much about perfection in the other areas. You can always make them better in your 2.0 release.

Monday, 14 July 2008

Your customers *do* know what they want

I had a conversation with an old school friend last night; and we had a brief discussion about entrepreneurship as it relates to IT. He said he wanted to get out of the daily grind, and thought it'd be great to come up with a product and sell it. He was a bit leery, still, not sure about whether he could really "make it".

Having read a number of Paul Graham's fantastic articles I thought the idea was great, but I reminded him of how important it was to make something that your customers actually want. At which point he stated: "I speak to my customers all the time and in my experience they don't know what they want"

At the time I didn't answer. I could see he had a point, but it wasn't the one I was trying to make, and the conversation moved on. But now I think about it I think that he was simply wrong.

What question?

When you ask a user what content they want on their site or ask about styling or layout, you'll frequently bump up against a puzzled look and a lot of hemming and hawing as they try to figure it out. They don't have an answer ready for that and they probably didn't think about it before you asked. Often they have no idea what they want.

Many is the project where the requirements change like a fickle breeze as the scope creeps inexorably onward. It's obvious your customers don't know what they want. Right?

or do they? is it possible you're asking them the wrong questions?

What language?

Since I moved into freelancing, I've been studying up a whole lot more on business. One thing I was surprised to learn is that my customers are actually pretty savvy. They know their business, and they know exactly why they came to you to build their website.

Your customer knows they want a website that promotes their brand to get more customers, or to cross-fertilise their customer-base with their other product-lines, or to act as a tool that is a more efficient/less-expensive way of selling their products.

What they want has to do with their *business* goals, not with the specifics of the technology that will be used to achieve that goal. They know exactly what the goal is and whether they have reached it - and it has nothing to do with font-sizes or table layouts, or whether the "order" button should be on the product list-page or on the product's details-page.

If you don't know what your customer wants - it's most likely because you haven't asked them the right questions. But it's not enough to change what you're asking - you also have to change how you're asking it.

Your customer lives in a different world to you. You live in IT-world, but they live in the world of their business. Your customer uses a language that is different to IT language. They speak in the language of business - with market-penetration, conversion-rates and revenue. They don't operate in the realm of splash-pages, JavaScript or three-column-layouts. Business has its own language, its own way of organising and expressing ideas, and its own way of determining whether or not a set of actions will acheive a required result.

More than your job's worth

If you don't speak the language of business, then you'd better learn quick! These are your customers, and you already know they can't speak IT, and don't have the time or patience to learn. So it's up to you to do the translation service.

After all, that's your job. It's why you get paid the "big bucks" right?

It's your job to learn how to ask questions in such a way that your customer understands what you're asking and why. To translate their requirements into IT-jargon, and your questions back into business-speak. To express their choices in such a way that it gives them enough information to actually make the decision that you are asking of them.

Do they want flash or XHTML? does it matter to them if you use table or CSS-layouts?

No - what they want to know is "will this process of online ordering appeal more to my customers than the other, or will it be too difficult to learn?", "will my customers be able to find my products on this website, or will they get confused and I lose a customer?", "how can I use this software to give me better tracking of purchasing so I can promote or cross-promote appropriately?".

Tell them that if you place the "order now" button on the product-listing page, it could speed up the ordering process, at the expense of creating visual clutter. Explain that new users might get confused by this, but that it could aid frequent users to get through the process quicker. It then becomes a business decision: ie "are we aiming this site at new customers, or our returning/frequent customers?" - which is a much more meaningful basis for their decision-making.

So...

Translate your requirement questions into their language. Explain how their choice will impact them and their business goals. Explain the benefits in a way that they will understand. Make sure what you ask them leads back to their goals - not yours, and you'll find that they actually have a very good grasp of what they want.

Thursday, 1 May 2008

Up or Out - the attitude for contractors

I've just read the article Up or Out on The Daily WTF. This is related to/based on Bruce Webster's article Wetware crisis: Dead Sea effect. They provide a fascinating look into the culture of the skilled IT workforce as it pertains to over-zealous attempts at retaining skilled workers.

So I got to thinking about how that applied to myself, and my past experience in IT. I've worked both as an employee and, recently, as a contractor - and I've certainly noticed the difference in attitude between the two.

I believe that contractors automatically have this "up or out" mentality. They know that their required skills are wide and varied, and that to keep ahead of the game, they need to keep their skill-level up. Staying in one job for a long term leads (eventually) to skill-stagnation. Not because the company no longer has anything to teach a contractor, but because of a case of diminishing returns.

As the article states, any skilled worker can learn a lot more by exposing themselves to new opportunities, than by sticking with a single firm. However, contractors seem to be far more aware of this fact than long-term employees (even the skilled ones). Contractors seem to have learned the lesson that current and diverse skills are marketable, and that in-house knowledge of a specific system, while beneficial to keeping a specific job, doesn't lead to expansion of a skill-set. This leads contractors to constantly push themselves to try new and different things to stay ahead of the game - to keep themselves a marketable commodity.

Employers don't seem to have kept up with this understanding. The majority of employers look poorly on hiring contractors (except for specific skills lacking in-house), compared with potential long-term employees. Contractors fall foul of the attitude mentioned in the article - the feeling that contractors are just "dating around" and aren't serious about their committment. This despite the fact that employers are no longer loyal towards employees (a job is no longer a career-for-life).

So are contracters flighty or are they simply being honest? We know that our skills improve by varied experience. Stagnating in a single shop isn't good for us *or* for the company - yet we constantly get the look-down-the-nose treatment as though we are far more unreliable than their indentured employees.

Workers and companies both benefit from fresh-blood. There is definitely a balance involved here - continual turnover can make it impossible to keep hold of institutional knowledge (which is why documenting procedures is so important!), but a company without fresh-blood will stagnate and lose market-share due to a lack of new and innovative insights coming into the company. Any firm can benefit from a good mix of the two. The fact that contractors have accepted and embraced this fact should be a sign of maturity, rather than flightiness.

I especially dislike that skeptical tone that comes with with the phrase "oh, so you're *not* interested in long-term employement". I feel I'm simply being realistic. But employers (and recruitment agencies) that employ the tone seem to imply I'm being disloyal... or committment-phobic. Neither of which is the case.

A few rare cases even seem to imply that I'm simply being greedy - shopping around for a better cash-deal, which is far from the case. While I'll take a higher salary over a lower one, I'm far more motivated by an interesting project and an opportunity to learn. That's why I'm in Rails rather than many other technologies that pay well. I would never take a higher salary just to do something that was mind-numbingly boring and, in my mind, dead-end. I'm just stoked that I get to work in a field that is cutting edge *and* well paid at the same time ;)

I have no problem with sticking around with a group that values me and that contributes to my own experience. My current contract (with SIRCA) has lasted more than a year. I fully believe that I have added value to this project. I also have learned a great deal. I have also had smaller contracts where I have provided some chunk of functionality or code-review - each of which added visible benefit to the site I worked upon, and also provided an opportunity for me to learn some new aspect of the technology I work with. This is an equal-footing relationship - where both parties receive value.

My previous employment has included roles in which I felt my input was not valued, and in which I had no mentor from which to learn. Effectively I was gaining no value from them, and they were not gaining full value from me either. I stuck around for a long time trying to help out with the project - and eventually left due to burn-out. In hindsight I should have left far earlier and moved on somewhere that I had an opportunity to grow, and that valued the insights I was able to bring. It would have been a better opportunity for me - and the company I was with could have employed somebody that they would have felt comfortable with - increasing their return-on-investment as well.

In a free market - there's no point in sticking with a relationship that is not valuable, or which provides value only to one party or the other. I won't leave for reasons of greed or lack of committment - I'll only leave if you and I are no longer gaining enough value for the professional relationship to be worthwhile.

If an employer doesn't feel that this is reasonable - then we need a good talk on the concept of "fairness".

Healthy economic relationships come about when both parties benefit. Anything else doesn't make economic sense. When the benefit to one or the other fades over time, then it should be understood that this will lead to the eventual ending of the relationship. It's simply a matter of good business practice - and shouldn't be looked on as "lack of committment" any more than moving to a cheaper/more efficient supplier.

</whinge> :)

Update: Bruce Webster has written a follow-on article: some thoughts on up or out which gives a few ideas on how to avoid developer-churn or the Dead-sea effect (and even counteract the thermocline of truth) by providing a non-managerial track for techies to follow. I completely agree. If there was a way "up" that didn't involve "out" I'd go for it!